We Keep Coding

my python Ray script runs on a single worker only

I am a new with Ray and after have read he documentation, I came up with a script that mimics what I want to do further with Ray. Here is my script:
import ray
import time
import h5py
#ray.remote
class Analysis:
def __init__(self):
self._file = h5py.File('./Data/Trajectories/MDANSE/apoferritin.h5')
def __getstate__(self):
print('I dump')
d = self.__dict__.copy()
del d['_file']
return d
def __setstate__(self,state):
self.__dict__ = state
self._file = h5py.File('./Data/Trajectories/MDANSE/apoferritin.h5')
def run_step(self,index):
time.sleep(5)
print('I run a step',index)
def combine(self,index):
print('I combine',index)
ray.init(num_cpus=4)
a = Analysis.remote()
obj_id = ray.put(a)
for i in range(100):
output = ray.get(a.run_step.remote(i))
My problem is that when I run this script it runs on a single worker as indicated by the Ray output whereas I would expect 4 workers to be fired. Would you know what is wrong with my script ?

Quoting from ray docs on actor
Methods called on different actors can execute in parallel, and methods called on the same actor are executed serially in the order that they are called.
Another issue with the above code is that ray.get is a blocking call.
I will suggest instantiating multiple actors and running the jobs, like
actors = [Analysis.remote() for i in range(num_cpus)]
outputs = []
for i in range(100):
outputs.append(actors[i % num_cpus].run_step.remote(i))
output = ray.get(outputs)

Python 3 asyncio with aioboto3 seems sequential

I am porting a simple python 3 script to AWS Lambda.
The script is simple: it gathers information from a dozen of S3 objects and returns the results.
The script used multiprocessing.Pool to gather all the files in parallel. Though multiprocessing cannot be used in an AWS Lambda environment since /dev/shm is missing.
So I thought instead of writing a dirty multiprocessing.Process / multiprocessing.Queue replacement, I would try asyncio instead.
I am using the latest version of aioboto3 (8.0.5) on Python 3.8.
My problem is that I cannot seem to gain any improvement between a naive sequential download of the files, and an asyncio event loop multiplexing the downloads.
Here are the two versions of my code.
import sys
import asyncio
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
import boto3
import aioboto3
BUCKET = 'some-bucket'
KEYS = [
'some/key/1',
[...]
'some/key/10',
]
async def download_aio():
"""Concurrent download of all objects from S3"""
async with aioboto3.client('s3') as s3:
objects = [s3.get_object(Bucket=BUCKET, Key=k) for k in KEYS]
objects = await asyncio.gather(*objects)
buffers = await asyncio.gather(*[o['Body'].read() for o in objects])
def download():
"""Sequentially download all objects from S3"""
s3 = boto3.client('s3')
for key in KEYS:
object = s3.get_object(Bucket=BUCKET, Key=key)
object['Body'].read()
def run_sequential():
download()
def run_concurrent():
loop = asyncio.get_event_loop()
#loop.set_default_executor(ProcessPoolExecutor(10))
#loop.set_default_executor(ThreadPoolExecutor(10))
loop.run_until_complete(download_aio())
The timing for both run_sequential() and run_concurrent() are quite similar (~3 seconds for a dozen of 10MB files).
I am convinced the concurrent version is not, for multiple reasons:
I tried switching to Process/ThreadPoolExecutor, and I the processes/threads spawned for the duration of the function, though they are doing nothing
The timing between sequential and concurrent is very close to the same, though my network interface is definitely not saturated, and the CPU is not bound either
The time taken by the concurrent version increases linearly with the number of files.
I am sure something is missing, but I just can't wrap my head around what.
Any ideas?

After loosing some hours trying to understand how to use aioboto3 correctly, I decided to just switch to my backup solution.
I ended up rolling my own naive version of multiprocessing.Pool for use within an AWS lambda environment.
If someone stumble across this thread in the future, here it is. It is far from perfect, but easy enough to replace multiprocessing.Pool as-is for my simple cases.
from multiprocessing import Process, Pipe
from multiprocessing.connection import wait
class Pool:
"""Naive implementation of a process pool with mp.Pool API.
This is useful since multiprocessing.Pool uses a Queue in /dev/shm, which
is not mounted in an AWS Lambda environment.
"""
def __init__(self, process_count=1):
assert process_count >= 1
self.process_count = process_count
#staticmethod
def wrap_pipe(pipe, index, func):
def wrapper(args):
try:
result = func(args)
except Exception as exc: # pylint: disable=broad-except
result = exc
pipe.send((index, result))
return wrapper
def __enter__(self):
return self
def __exit__(self, exc_type, exc_value, exc_traceback):
pass
def map(self, function, arguments):
pending = list(enumerate(arguments))
running = []
finished = [None] * len(pending)
while pending or running:
# Fill the running queue with new jobs
while len(running) < self.process_count:
if not pending:
break
index, args = pending.pop(0)
pipe_parent, pipe_child = Pipe(False)
process = Process(
target=Pool.wrap_pipe(pipe_child, index, function),
args=(args, ))
process.start()
running.append((index, process, pipe_parent))
# Wait for jobs to finish
for pipe in wait(list(map(lambda t: t[2], running))):
index, result = pipe.recv()
# Remove the finished job from the running list
running = list(filter(lambda x: x[0] != index, running))
# Add the result to the finished list
finished[index] = result
return finished

it's 1.5 years later and aioboto3 is still not well documented or supported.
The multithreading option is good. but AIO is an easier and more clear implementation
I don't actually know what's wrong with your AIO code. It's even not running now because of the updates I guess. but using aiobotocore this code worked. my test was with 100 images. in the sequential code, it takes 8 sec. in average. in IO it was less than 2.
with 1000 images it was 17 sec.
import asyncio
from aiobotocore.session import get_session
async def download_aio(s3,bucket,file_name):
o = await s3.get_object(Bucket=bucket, Key=file_name)
x = await o['Body'].read()
async def run_concurrent():
tasks =[]
session = get_session()
async with session.create_client('s3') as s3:
for k in KEYS[:100]:
tasks.append(asyncio.ensure_future(get_object(s3,BUCKET,k)))
await asyncio.gather(*tasks)

Python wait for x seconds without sleeping program? [duplicate]

I'm trying to run 2 functions at the same time.
def func1():
print('Working')
def func2():
print('Working')
func1()
func2()
Does anyone know how to do this?

Do this:
from threading import Thread
def func1():
print('Working')
def func2():
print("Working")
if __name__ == '__main__':
Thread(target = func1).start()
Thread(target = func2).start()

The answer about threading is good, but you need to be a bit more specific about what you want to do.
If you have two functions that both use a lot of CPU, threading (in CPython) will probably get you nowhere. Then you might want to have a look at the multiprocessing module or possibly you might want to use jython/IronPython.
If CPU-bound performance is the reason, you could even implement things in (non-threaded) C and get a much bigger speedup than doing two parallel things in python.
Without more information, it isn't easy to come up with a good answer.

This can be done elegantly with Ray, a system that allows you to easily parallelize and distribute your Python code.
To parallelize your example, you'd need to define your functions with the #ray.remote decorator, and then invoke them with .remote.
import ray
ray.init()
# Define functions you want to execute in parallel using
# the ray.remote decorator.
#ray.remote
def func1():
print("Working")
#ray.remote
def func2():
print("Working")
# Execute func1 and func2 in parallel.
ray.get([func1.remote(), func2.remote()])
If func1() and func2() return results, you need to rewrite the above code a bit, by replacing ray.get([func1.remote(), func2.remote()]) with:
ret_id1 = func1.remote()
ret_id2 = func1.remote()
ret1, ret2 = ray.get([ret_id1, ret_id2])
There are a number of advantages of using Ray over the multiprocessing module or using multithreading. In particular, the same code will run on a single machine as well as on a cluster of machines.
For more advantages of Ray see this related post.

One option, that looks like it makes two functions run at the same
time, is using the threading module (example in this answer).
However, it has a small delay, as an Official Python Documentation
page describes. A better module to try using is multiprocessing.
Also, there's other Python modules that can be used for asynchronous execution (two pieces of code working at the same time). For some information about them and help to choose one, you can read this Stack Overflow question.
Comment from another user about the threading module
He might want to know that because of the Global Interpreter Lock
they will not execute at the exact same time even if the machine in
question has multiple CPUs. wiki.python.org/moin/GlobalInterpreterLock
– Jonas Elfström Jun 2 '10 at 11:39
Quote from the Documentation about threading module not working
CPython implementation detail: In CPython, due to the Global Interpreter
Lock, only one thread can execute Python code at once (even though
certain performance-oriented libraries might overcome this limitation).
If you want your application to make better use of the computational resources of multi-core machines, you are advised to use multiprocessing or concurrent.futures.ProcessPoolExecutor.
However, threading is still an appropriate model if you
want to run multiple I/O-bound tasks simultaneously.

The thread module does work simultaneously unlike multiprocess, but the timing is a bit off. The code below prints a "1" and a "2". These are called by different functions respectively. I did notice that when printed to the console, they would have slightly different timings.
from threading import Thread
def one():
while(1 == num):
print("1")
time.sleep(2)
def two():
while(1 == num):
print("2")
time.sleep(2)
p1 = Thread(target = one)
p2 = Thread(target = two)
p1.start()
p2.start()
Output: (Note the space is for the wait in between printing)
1
2
2
1
12
21
12
1
2
Not sure if there is a way to correct this, or if it matters at all. Just something I noticed.

Try this
from threading import Thread
def fun1():
print("Working1")
def fun2():
print("Working2")
t1 = Thread(target=fun1)
t2 = Thread(target=fun2)
t1.start()
t2.start()

In case you also want to wait until both functions have been completed:
from threading import Thread
def func1():
print 'Working'
def func2():
print 'Working'
# Define the threads and put them in an array
threads = [
Thread(target = self.func1),
Thread(target = self.func2)
]
# Func1 and Func2 run in separate threads
for thread in threads:
thread.start()
# Wait until both Func1 and Func2 have finished
for thread in threads:
thread.join()

Another approach to run multiple functions concurrently in python is using asyncio that I couldn't see within the answers.
import asyncio
async def func1():
for _ in range(5):
print(func1.__name__)
await asyncio.sleep(0) # switches tasks every iteration.
async def func2():
for _ in range(5):
print(func2.__name__)
await asyncio.sleep(0)
tasks = [func1(), func2()]
await asyncio.gather(*tasks)
Out:
func1
func2
func1
func2
func1
func2
func1
func2
func1
func2
[NOTE]:
The above asyncio syntax is valid on python 3.7 and later
multiprocessing vs multithreading vs asyncio

This code below can run 2 functions parallelly:
from multiprocessing import Process
def test1():
print("Test1")
def test2():
print("Test2")
if __name__ == "__main__":
process1 = Process(target=test1)
process2 = Process(target=test2)
process1.start()
process2.start()
process1.join()
process2.join()
Result:
Test1
Test2
And, these 2 sets of code below can run 2 functions concurrently:
from threading import Thread
def test1():
print("Test1")
def test2():
print("Test2")
thread1 = Thread(target=test1)
thread2 = Thread(target=test2)
thread1.start()
thread2.start()
thread1.join()
thread2.join()
from operator import methodcaller
from multiprocessing.pool import ThreadPool
def test1():
print("Test1")
def test2():
print("Test2")
caller = methodcaller("__call__")
ThreadPool().map(caller, [test1, test2])
Result:
Test1
Test2
And, this code below can run 2 async functions concurrently and asynchronously:
import asyncio
async def test1():
print("Test1")
async def test2():
print("Test2")
async def call_tests():
await asyncio.gather(test1(), test2())
asyncio.run(call_tests())
Result:
Test1
Test2

I think what you are trying to convey can be achieved through multiprocessing. However if you want to do it through threads you can do this.
This might help
from threading import Thread
import time
def func1():
print 'Working'
time.sleep(2)
def func2():
print 'Working'
time.sleep(2)
th = Thread(target=func1)
th.start()
th1=Thread(target=func2)
th1.start()

test using APscheduler:
from apscheduler.schedulers.background import BackgroundScheduler
import datetime
dt = datetime.datetime
Future = dt.now() + datetime.timedelta(milliseconds=2550) # 2.55 seconds from now testing start accuracy
def myjob1():
print('started job 1: ' + str(dt.now())[:-3]) # timed to millisecond because thats where it varies
time.sleep(5)
print('job 1 half at: ' + str(dt.now())[:-3])
time.sleep(5)
print('job 1 done at: ' + str(dt.now())[:-3])
def myjob2():
print('started job 2: ' + str(dt.now())[:-3])
time.sleep(5)
print('job 2 half at: ' + str(dt.now())[:-3])
time.sleep(5)
print('job 2 done at: ' + str(dt.now())[:-3])
print(' current time: ' + str(dt.now())[:-3])
print(' do job 1 at: ' + str(Future)[:-3] + '''
do job 2 at: ''' + str(Future)[:-3])
sched.add_job(myjob1, 'date', run_date=Future)
sched.add_job(myjob2, 'date', run_date=Future)
i got these results. which proves they are running at the same time.
current time: 2020-12-15 01:54:26.526
do job 1 at: 2020-12-15 01:54:29.072 # i figure these both say .072 because its 1 line of print code
do job 2 at: 2020-12-15 01:54:29.072
started job 2: 2020-12-15 01:54:29.075 # notice job 2 started before job 1, but code calls job 1 first.
started job 1: 2020-12-15 01:54:29.076
job 2 half at: 2020-12-15 01:54:34.077 # halfway point on each job completed same time accurate to the millisecond
job 1 half at: 2020-12-15 01:54:34.077
job 1 done at: 2020-12-15 01:54:39.078 # job 1 finished first. making it .004 seconds faster.
job 2 done at: 2020-12-15 01:54:39.091 # job 2 was .002 seconds faster the second test

I might be wrong but:
with this piece of code:
def function_sleep():
time.sleep(5)
start_time = time.time()
p1=Process(target=function_sleep)
p2=Process(target=function_sleep)
p1.start()
p2.start()
end_time = time.time()
I took the time and I would expect to get 5/6 seconds, while it always takes the double of the argument passed to the function sleep (10 seconds in this case).
What's the matter?
Sorry guys, as mentioned in the previous comment, the "join()" need to be called.
That's very important!

How get value from function which is executed by Thread?

I have main_script.py which import scripts which get data from webpages. I want do this by use multithreading. I came up with this solution, but it does not work:
main_script:
import script1
temp_path = ''
thread1 = threading.Thread(target=script1.Main,
name='Script1',
args=(temp_path, ))
thread1.start()
thread1.join()
script1:
class Main:
def __init__()
def some_func()
def some_func2()
def __main__():
some_func()
some_func2()
return callback
Now only 1 way I know to get value of callback from script1 to main_script is:
main_script:
import script1
temp_path = ''
# make instance of class with temp_path
inst_script1 = script1.Main(temp_path)
print("instance1:")
print(inst_script1.callback)
It's works but then I run instances of scripts one-by-one, no concurrently.
Anybody has any idea how handle that? :)

First off if you are using threading in Python make sure you read: https://docs.python.org/2/glossary.html#term-global-interpreter-lock. Unless you are using C modules or a lot of I/O you won't see the scripts run concurrently. Generally speaking, multiprocessing.pool is a better approach.
If you are certain we want threads rather then processes you can use a mutable variable to store the result. For example, a dictionary which keeps track of the result of each thread.
result = {}
def test(val, name, target):
target[name] = val * 4
temp_path = 'ASD'
thread1 = threading.Thread(target=test,
name='Script1',
args=(temp_path, 'A', result))
thread1.start()
thread1.join()
print (result)

Thanks for response. Yes, I readed about GIL, but it's doesn't make me any problem yet. Generally I solve my problem, because I find solution on other website. Code like this now:
Main_script:
import queue
import script1
import script2
queue_callbacks = queue.Queue()
threads_list = list()
temp_path1 = ''
thread1 = threading.Thread(target= lambda q, arg1: q.put(Script1.Main(arg1)),
name='Script1',
args=(queue_callbacks, temp_path1, ))
thread1.start()
temp_path2 = ''
thread2 = threading.Thread(target= lambda q, arg1: q.put(Script2.Main(arg1)),
name='Script2',
args=(queue_callbacks, temp_path2, ))
thread2.start()
for t in threads_list:
t.join()
while not kolejka_callbacks.empty():
result = queue_callbacks.get()
callbacks.append({"service": result.service, "callback": result.callback, "error": result.error})
And this works fine. Now I have other problem, because I want this to work in big scale, where I have a hundreds of scripts and handle this by e.q. 5 threads.
In general, is there any limit to the number of threads running at any one time?

Using Multiprocessing with Modules

I am writing a module such that in one function I want to use the Pool function from the multiprocessing library in Python 3.6. I have done some research on the problem and the it seems that you cannot use if __name__=="__main__" as the code is not being run from main. I have also noticed that the python pool processes get initialized in my task manager but essentially are stuck.
So for example:
class myClass()
...
lots of different functions here
...
def multiprocessFunc()
do stuff in here
def funcThatCallsMultiprocessFunc()
array=[array of filenames to be called]
if __name__=="__main__":
p = Pool(processes=20)
p.map_async(multiprocessFunc,array)
I tried to remove the if __name__=="__main__" part but still no dice. any help would appreciated.

It seems to me that your have just missed out a self. from your code. I should think this will work:
class myClass():
...
# lots of different functions here
...
def multiprocessFunc(self, file):
# do stuff in here
def funcThatCallsMultiprocessFunc(self):
array = [array of filenames to be called]
p = Pool(processes=20)
p.map_async(self.multiprocessFunc, array) #added self. here
Now having done some experiments, I see that map_async could take quite some time to start up (I think because multiprocessing creates processes) and any test code might call funcThatCallsMultiprocessFunc and then quit before the Pool has got started.
In my tests I had to wait for over 10 seconds after funcThatCallsMultiprocessFunc before calls to multiprocessFunc started. But once started, they seemed to run just fine.
This is the actual code I've used:
MyClass.py
from multiprocessing import Pool
import time
import string
class myClass():
def __init__(self):
self.result = None
def multiprocessFunc(self, f):
time.sleep(1)
print(f)
return f
def funcThatCallsMultiprocessFunc(self):
array = [c for c in string.ascii_lowercase]
print(array)
p = Pool(processes=20)
p.map_async(self.multiprocessFunc, array, callback=self.done)
p.close()
def done(self, arg):
self.result = 'Done'
print('done', arg)
Run.py
from MyClass import myClass
import time
def main():
c = myClass()
c.funcThatCallsMultiprocessFunc()
for i in range(30):
print(i, c.result)
time.sleep(1)
if __name__=="__main__":
main()

The if __name__=='__main__' construct is an import protection. You want to use it, to stop multiprocessing from running your setup on import.
In your case, you can leave out this protection in the class setup. Be sure to protect the execution points of the class in the calling file like this:
def apply_async_with_callback():
pool = mp.Pool(processes=30)
for i in range(z):
pool.apply_async(parallel_function, args = (i,x,y, ), callback = callback_function)
pool.close()
pool.join()
print "Multiprocessing done!"
if __name__ == '__main__':
apply_async_with_callback()