We Keep Coding

python multiprocessing map function

I encountered a problem while writing the python code with a multiprocessing map function. The minimum code to reproduce the problem is like
import multiprocessing as mp
if __name__ == '__main__':
def f(x):
return x*x
num_workers = 2
with mp.Pool(num_workers) as p:
print(p.map(f, [1,2,3]))
If one runs this piece of code, I got the error message
AttributeError: Can't get attribute 'f' on <module '__mp_main__' from 'main.py'>
However, If I move f-function outside the main function, i.e.
import multiprocessing as mp
def f(x):
return x*x
if __name__ == '__main__':
num_workers = 2
with mp.Pool(num_workers) as p:
print(p.map(f, [1,2,3]))
It works this time. I am wondering what's the difference between them and how can I get an error in the first version. Thanks in advance.

Depending on your operating system, sub-processes will either be forked or spawned. macOS, for example, will spawn whereas Windows will fork.
You can enforce forking but you need to fully understand the implications of doing so.
For this specific question a workaround could be implemented thus:
import multiprocessing as mp
from multiprocessing import set_start_method
if __name__ == '__main__':
def f(x):
return x*x
set_start_method('fork')
num_workers = 2
with mp.Pool(num_workers) as p:
print(p.map(f, [1,2,3]))

This will vary between operating systems, but the basic reason is that this line of code
if __name__ == '__main__':
is telling the Python interpreter to only include anything in this code section in the main process when run as a script - it won't be included in any sub process, nor will it appear if you import it as a module. So when you do this
import multiprocessing as mp
if __name__ == '__main__':
def f(x):
return x*x
num_workers = 2
with mp.Pool(num_workers) as p:
print(p.map(f, [1,2,3]))
any sub processes created by p.map will not have the definition of function f

Multiprocessing with python nothing happens

I am trying to use Multiprocessing to scrape web pages but when I launch my program, nothing happened so I tried the (very) simple example and also nothing happens. So the problem seems to be with multiprocessing :
from multiprocessing import Pool
def f(x):
return x*x
if __name__ == '__main__':
with Pool(5) as p:
print(p.map(f, [1, 2, 3]))
Any ideas ?
I am on macbookPro with JupyterLab 2.2.6
Thx

Multiprocessing hanging in Spyder

I have been trying the pool.map() multiprocessing with python3 and not matter how I simplify my function it hangs and shows that the code is still running and gives no results.
I am using Windows.Here is my code:
import multiprocessing as mp
def f(x):
return x + 1
if __name__ == '__main__':
with mp.Pool() as pool:
print(pool.map(f, range(10)))
Can anyone tell me how I can solve this problem?
Thanks!

Python Multiprocessing Pool Class won't work

I am trying to use multiprocessing on a different problem but I can't get it to work. To make sure I'm using the Pool class correctly, I made the following simpler problem but even that won't work. What am I doing wrong here?
from multiprocessing import Pool
def square(x):
sq = x**2
return sq
def main():
x1 = [1,2,3,4]
pool = Pool()
result = pool.map( square, x1 )
print(result)
if __name__ == '__main__': main()
The computer just seems to run forever and I need to close and restart the IPython shell before I can do anything.

I figured out what was wrong. I named the script "multiprocessing.py" which is the name of the module that was being imported. This resulted in the script attempting to import itself instead of the actual module.

Python Pool Silently Crashes When Defined in IDLE

I am trying to use the multiprocessing packages for python. However, when I try to define a pool in IDLE it silently crashes (no error message or traceback, IDLE just closes). On the other hand, this same script has no problem running when executed from the terminal. What gives? I am using python 2.7 on Ubuntu 12.04.
import multiprocessing
from multiprocessing import Pool
def myfunc(x):
return x*x
cpu_count = int(multiprocessing.cpu_count() - 1)
pool = Pool(processes = cpu_count) #Crashes here in IDLE
resultlist = pool.map(myfunc, range(10))
pool.close()
print(resultlist)

The problem is your code: it is missing the if __name__ == '__main__': guard clause that is an essential part of every working example in the multiprocessing doc chapter, as it keeps each subprocess from running the start and finish code. Running the following in Idle (or without Idle, in a console)
import multiprocessing
from multiprocessing import Pool
def myfunc(x):
return x*x
if __name__ == '__main__':
cpu_count = int(multiprocessing.cpu_count() - 1)
pool = Pool(processes = cpu_count) #Crashes here in IDLE
resultlist = pool.map(myfunc, range(10))
pool.close()
print(resultlist)
almost immediately prints
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
If you run your original code in a console (or with Idle started in the console), you will see an endless stream of error messages as each process starts up more processes.
Edit: the above behavior is on Windows 10
I suggest you ignore opinions and decide for yourself what tools you want to use, based on current and real facts. Try to use the latest bugfix release of whatever Python version you use. For Idle in particular, there have been many fixes in the last 2 years after a couple of years or so of stagnation.