We Keep Coding

How to initialize parallel independent process within function?

Sorry, if headline is strange. Let me explain.
Let's say there is handler.py:
import funcs
import requests
def initialize_calculate(data):
check_data(data)
funcs.calculate(data) # takes a lot of time like 30 minutes
print('Calculation launched')
requests.get('hostname', params={'func':'calculate', 'status':'launched'})
and here is funcs.py:
import requests
def calculate(data):
result = make_calculations(data)
requests.get('hostname',params={'func':'calculate', 'status':'finished', 'result':result})
So what I want is that handler can initialize another function no matter where, but doesn't wait until it ends, because I want to notify client-side that process is started, and when it's done this process itself will send result when it's finished.
How can I launch independent process with function calculate from initialize_calculate?
I want to know If it's possible without non-native libraries or frameworks.

If you don't wan't to use a 3rd-party lib like daemonocle implementing a "well-behaved" Unix-Daemon, you could
use subprocess.Popen() to create an independent process. Another option would be to modify multiprocessing.Process to prevent auto-joining of the child when the parent exits.
subprocess.Popen()
With subprocess.Popen() you start the new process with specifying commands and arguments like manually from terminal. This means you need to make funcs.py or another file a top-level script which parses string-arguments from stdin and then calls funcs.calculate() with these arguments.
I boiled your example down to the essence so we don't have to read too much code.
funcs.py
#!/usr/bin/env python3
# UNIX: enable executable from terminal with: chmod +x filename
import os
import sys
import time
import psutil # 3rd party for demo
def print_msg(msg):
print(f"[{time.ctime()}, pid: {os.getpid()}] --- {msg}")
def calculate(data, *args):
print_msg(f"parent pid: {psutil.Process().parent().pid}, start calculate()")
for _ in range(int(500e6)):
pass
print_msg(f"parent pid: {psutil.Process().parent().pid}, end calculate()")
if __name__ == '__main__':
if len(sys.argv) > 1:
calculate(*sys.argv[1:])
subp_main.py
#!/usr/bin/env python3
# UNIX: enable executable from terminal with: chmod +x filename
if __name__ == '__main__':
import time
import logging
import subprocess
import multiprocessing as mp
import funcs
mp.log_to_stderr(logging.DEBUG)
filename = funcs.__file__
data = ("data", 42)
# in case filename is an executable you don't need "python" before `filename`:
subprocess.Popen(args=["python", filename, *[str(arg) for arg in data]])
time.sleep(1) # keep parent alive a bit longer for demo
funcs.print_msg(f"exiting")
And important for testing, run from terminal, e.g. not PyCharm-Run, because it won't show what the child prints. In the last line below you see the child process' parent-id changed to 1 because the child got adopted by systemd (Ubuntu) after the parent exited.
$> ./subp_main.py
[Fri Oct 23 20:14:44 2020, pid: 28650] --- parent pid: 28649, start calculate()
[Fri Oct 23 20:14:45 2020, pid: 28649] --- exiting
[INFO/MainProcess] process shutting down
[DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0
[DEBUG/MainProcess] running the remaining "atexit" finalizers
$> [Fri Oct 23 20:14:54 2020, pid: 28650] --- parent pid: 1, end calculate()
class OrphanProcess(multiprocessing.Process)
If you search for something more convenient, well you can't use the high-level multiprocessing.Process as is, because it doesn't let the parent process exit before the child, as you asked for. Regular child-processes are either joined (awaited) or terminated (if you set the daemon-flag for Process) when the parent shuts down. This still happens within Python. Note that the daemon-flag doesn't make a process a Unix-Daemon. The naming is a somewhat frequent source of confusion.
I subclassed multiprocessing.Process to switch the auto-joining off and spend some time with the source and observing if zombies might become an issue. Because the modification turns off automatic joining in the parent, I recommend using "forkserver" as start-method for new processes on Unix (always a good idea if the parent is already multi-threaded) to prevent zombie-children from sticking around as long the parent is still running. When the parent process terminates, its child-zombies get eventually reaped by systemd/init. Running multiprocessing.log_to_stderr() shows everything shutting down cleanly, so nothing seems broken so far.
Consider this approach experimental, but it's probably a lot safer than using raw os.fork() to re-invent part of the extensive multiprocessing machinery, just to add this one feature. For error-handling in the child, write a try-except block and log to file.
orphan.py
import multiprocessing.util
import multiprocessing.process as mpp
import multiprocessing as mp
__all__ = ['OrphanProcess']
class OrphanProcess(mp.Process):
"""Process which won't be joined by parent on parent shutdown."""
def start(self):
super().start()
mpp._children.discard(self)
def __del__(self):
# Finalizer won't `.join()` the child because we discarded it,
# so here last chance to reap a possible zombie from within Python.
# Otherwise systemd/init will reap eventually.
self.join(0)
orph_main.py
#!/usr/bin/env python3
# UNIX: enable executable from terminal with: chmod +x filename
if __name__ == '__main__':
import time
import logging
import multiprocessing as mp
from orphan import OrphanProcess
from funcs import print_msg, calculate
mp.set_start_method("forkserver")
mp.log_to_stderr(logging.DEBUG)
p = OrphanProcess(target=calculate, args=("data", 42))
p.start()
time.sleep(1)
print_msg(f"exiting")
Again test from terminal to get the child print to stdout. When the shell appears to be hanging after everything was printed over the second prompt, hit enter to get a new prompt. The parent-id stays the same here because the parent, from the OS-point of view, is the forkserver-process, not the initial main-process for orph_main.py.
$> ./orph_main.py
[INFO/MainProcess] created temp directory /tmp/pymp-bd75vnol
[INFO/OrphanProcess-1] child process calling self.run()
[Fri Oct 23 21:18:29 2020, pid: 30998] --- parent pid: 30997, start calculate()
[Fri Oct 23 21:18:30 2020, pid: 30995] --- exiting
[INFO/MainProcess] process shutting down
[DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0
[DEBUG/MainProcess] running the remaining "atexit" finalizers
$> [Fri Oct 23 21:18:38 2020, pid: 30998] --- parent pid: 30997, end calculate()
[INFO/OrphanProcess-1] process shutting down
[DEBUG/OrphanProcess-1] running all "atexit" finalizers with priority >= 0
[DEBUG/OrphanProcess-1] running the remaining "atexit" finalizers
[INFO/OrphanProcess-1] process exiting with exitcode 0

you may use Process class from multiprocessing module to do that.
Here is an example:
from multiprocessing import Process
import requests
def calculate(data):
result = make_calculations(data)
requests.get('hostname',params={'func':'calculate', 'status':'finished', 'result':result})
def initialize_calculate(data):
check_data(data)
p = Process(target=calculate, args=(data,))
p.start()
print('Calculation launched')
requests.get('hostname', params={'func':'calculate', 'status':'launched'})

Why do main- and child-process not exit after exception in main?

#coding=utf8
from multiprocessing import Process
from time import sleep
import os
def foo():
print("foo")
for i in range(11000):
sleep(1)
print("{}: {}".format(os.getpid(), i))
if __name__ == "__main__":
p = Process(target=foo)
#p.daemon = True
p.start()
print("parent: {}".format(os.getpid()))
sleep(20)
raise Exception("invalid")
An exception is raised in the main-process, but child- and main-process keep running. Why?

When the MainProcess is shutting down, non-daemonic child processes are simply joined. This happens in _exit_function() which is registered with atexit.register(_exit_function). You can look it up in multiprocessing.util.py if you are curious.
You can also insert multiprocessing.log_to_stderr(logging.DEBUG) before you start your process to see the log messages:
parent: 9670
foo
[INFO/Process-1] child process calling self.run()
9675: 0
9675: 1
...
9675: 18
Traceback (most recent call last):
File "/home/...", line 26, in <module>
raise Exception("invalid")
Exception: invalid
[INFO/MainProcess] process shutting down
[DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0
[INFO/MainProcess] calling join() for process Process-1
9675: 19

Multiprocessing.Queue with hugh data causes _wait_for_tstate_lock

An Exception is raised in threading._wait_for_tstate_lock when I transfere hugh data between a Process and a Thread via multiprocessing.Queue.
My minimal working example looks a bit complex first - sorry. I will explain. The original application loads a lot of (not so important) files into RAM. This is done in a separate process to save ressources. The main gui thread shouldn't freez.
The GUI start a separate Thread to prevent the gui event loop from freezing.
This separate Thread then starts one Process which should does the work.
a) This Thread instanciates a multiprocess.Queue (be aware that this is a multiprocessing and not threading!)
b) This is givin to the Process for sharing data from Process back to the Thread.
The Process does some work (3 steps) and .put() the result into the multiprocessing.Queue.
When the Process ends the Thread takes over again and collect the data from the Queue, store it to its own attribute MyThread.result.
The Thread tells the GUI main loop/thread to call a callback function if it has time for.
The callback function (MyWindow::callback_thread_finished()) get the results from MyWindow.thread.result.
The problem is if the data put to the Queue is to big something happen I don't understand - the MyThread never ends. I have to cancle the application via Strg+C.
I got some hints from the docs. But my problem is I did not fully understand the documentation. But I have the feeling that the key of my problems can be found there.
Please see the two red boxex in "Pipes and Queues" (Python 3.5 docs).
That is the full output
MyWindow::do_start()
Running MyThread...
Running MyProcess...
MyProcess stoppd.
^CProcess MyProcess-1:
Exception ignored in: <module 'threading' from '/usr/lib/python3.5/threading.py'>
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 1288, in _shutdown
t.join()
File "/usr/lib/python3.5/threading.py", line 1054, in join
self._wait_for_tstate_lock()
File "/usr/lib/python3.5/threading.py", line 1070, in _wait_for_tstate_lock
elif lock.acquire(block, timeout):
KeyboardInterrupt
Traceback (most recent call last):
File "/usr/lib/python3.5/multiprocessing/process.py", line 252, in _bootstrap
util._exit_function()
File "/usr/lib/python3.5/multiprocessing/util.py", line 314, in _exit_function
_run_finalizers()
File "/usr/lib/python3.5/multiprocessing/util.py", line 254, in _run_finalizers
finalizer()
File "/usr/lib/python3.5/multiprocessing/util.py", line 186, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/usr/lib/python3.5/multiprocessing/queues.py", line 198, in _finalize_join
thread.join()
File "/usr/lib/python3.5/threading.py", line 1054, in join
self._wait_for_tstate_lock()
File "/usr/lib/python3.5/threading.py", line 1070, in _wait_for_tstate_lock
elif lock.acquire(block, timeout):
KeyboardInterrupt
This is the minimal working example
#!/usr/bin/env python3
import multiprocessing
import threading
import time
import gi
gi.require_version('Gtk', '3.0')
from gi.repository import Gtk
from gi.repository import GLib
class MyThread (threading.Thread):
"""This thread just starts the process."""
def __init__(self, callback):
threading.Thread.__init__(self)
self._callback = callback
def run(self):
print('Running MyThread...')
self.result = []
queue = multiprocessing.Queue()
process = MyProcess(queue)
process.start()
process.join()
while not queue.empty():
process_result = queue.get()
self.result.append(process_result)
print('MyThread stoppd.')
GLib.idle_add(self._callback)
class MyProcess (multiprocessing.Process):
def __init__(self, queue):
multiprocessing.Process.__init__(self)
self.queue = queue
def run(self):
print('Running MyProcess...')
for i in range(3):
self.queue.put((i, 'x'*102048))
print('MyProcess stoppd.')
class MyWindow (Gtk.Window):
def __init__(self):
Gtk.Window.__init__(self)
self.connect('destroy', Gtk.main_quit)
GLib.timeout_add(2000, self.do_start)
def do_start(self):
print('MyWindow::do_start()')
# The process need to be started from a separate thread
# to prevent the main thread (which is the gui main loop)
# from freezing while waiting for the process result.
self.thread = MyThread(self.callback_thread_finished)
self.thread.start()
def callback_thread_finished(self):
result = self.thread.result
for r in result:
print('{} {}...'.format(r[0], r[1][:10]))
if __name__ == '__main__':
win = MyWindow()
win.show_all()
Gtk.main()
Possible duplicate but quite different and IMO without an answer for my situation: Thread._wait_for_tstate_lock() never returns.
Workaround
Using a Manager by modifing line 22 to queue = multiprocessing.Manager().Queue() solve the problem. But I don't know why. My intention of this question is to understand the things behind and not only to make my code work. Even I don't really know what a Manager() is and if it has other (problem causing) implications.

According to the second warning box in the documentation you are linking to you can get a deadlock when you join a process before processing all items in the queue. So starting the process and immediately joining it and then processing the items in the queue is the wrong order of steps. You have to start the process, then receive the items, and then only when all items are received you can call the join method. Define some sentinel value to signal that the process is finished sending data through the queue. None for example if that can't be a regular value you expect from the process.
class MyThread(threading.Thread):
"""This thread just starts the process."""
def __init__(self, callback):
threading.Thread.__init__(self)
self._callback = callback
self.result = []
def run(self):
print('Running MyThread...')
queue = multiprocessing.Queue()
process = MyProcess(queue)
process.start()
while True:
process_result = queue.get()
if process_result is None:
break
self.result.append(process_result)
process.join()
print('MyThread stoppd.')
GLib.idle_add(self._callback)
class MyProcess(multiprocessing.Process):
def __init__(self, queue):
multiprocessing.Process.__init__(self)
self.queue = queue
def run(self):
print('Running MyProcess...')
for i in range(3):
self.queue.put((i, 'x' * 102048))
self.queue.put(None)
print('MyProcess stoppd.')

The documentation in question
reads:
Warning
As mentioned above, if a child process has put items on a queue (and it has not used JoinableQueue.cancel_join_thread), then that process will not terminate until all buffered items have been flushed to the pipe.
This means that if you try joining that process you may get a deadlock unless you are sure that all items which have been put on the queue have been consumed. Similarly, if the child process is non-daemonic then the parent process may hang on exit when it tries to join all its non-daemonic children.
Note that a queue created using a manager does not have this issue. See Programming guidelines.
This is supplementary to the accepted answer, but the edit queue is full.

multiprocessing.Queue fails intermittently. Bug in Python?

Python's multiprocessing.Queuefails intermittently, and I don't know why. Is this a bug in Python or my script?
Minimal failing script
import multiprocessing
import time
import logging
import multiprocessing.util
multiprocessing.util.log_to_stderr(level=logging.DEBUG)
queue = multiprocessing.Queue(maxsize=10)
def worker(queue):
queue.put('abcdefghijklmnop')
# "Indicate that no more data will be put on this queue by the
# current process." --Documentation
# time.sleep(0.01)
queue.close()
proc = multiprocessing.Process(target=worker, args=(queue,))
proc.start()
# "Indicate that no more data will be put on this queue by the current
# process." --Documentation
# time.sleep(0.01)
queue.close()
proc.join()
I am testing this in CPython 3.6.6 in Debian. It also fails with docker python:3.7.0-alpine.
docker run --rm -v "${PWD}/test.py:/test.py" \
python:3-alpine python3 /test.py
The above script sometimes fails with a BrokenPipeError.
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/queues.py", line 240, in _feed
send_bytes(obj)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/usr/lib/python3.6/multiprocessing/connection.py", line 404, in _send_bytes
self._send(header + buf)
File "/usr/lib/python3.6/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Test harness
Because this is intermittent, I wrote a shell script to call it many times and count the failures.
#!/bin/sh
total=10
successes=0
for i in `seq ${total}`
do
if ! docker run --rm -v "${PWD}/test.py:/test.py" python:3-alpine \
python3 test.py 2>&1 \
| grep --silent BrokenPipeError
then
successes=$(expr ${successes} + 1)
fi
done
python3 -c "print(${successes} / ${total})"
This usually shows some fraction, maybe 0.2 indicating intermittent failures.
Timing adjustments
If I insert time.sleep(0.01) before either queue.close(), it works consistently. I noticed in the source code that writing happens in its own thread. I think if the writing thread is still trying to write data and all of the other threads close the queue, then it causes the error.
Debug logs
By uncommenting the first few lines, I can trace the execution for failures and successes.
Failure:
[DEBUG/MainProcess] created semlock with handle 140480257941504
[DEBUG/MainProcess] created semlock with handle 140480257937408
[DEBUG/MainProcess] created semlock with handle 140480257933312
[DEBUG/MainProcess] Queue._after_fork()
[DEBUG/Process-1] Queue._after_fork()
[INFO/Process-1] child process calling self.run()
[DEBUG/Process-1] Queue._start_thread()
[DEBUG/Process-1] doing self._thread.start()
[DEBUG/Process-1] starting thread to feed data to pipe
[DEBUG/Process-1] ... done self._thread.start()
[DEBUG/Process-1] telling queue thread to quit
[INFO/Process-1] process shutting down
[DEBUG/Process-1] running all "atexit" finalizers with priority >= 0
[DEBUG/Process-1] running the remaining "atexit" finalizers
[DEBUG/Process-1] joining queue thread
Traceback (most recent call last):
File "/usr/lib/python3.7/multiprocessing/queues.py", line 242, in _feed
send_bytes(obj)
File "/usr/lib/python3.7/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/usr/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
self._send(header + buf)
File "/usr/lib/python3.7/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
[DEBUG/Process-1] feeder thread got sentinel -- exiting
[DEBUG/Process-1] ... queue thread joined
[INFO/Process-1] process exiting with exitcode 0
[INFO/MainProcess] process shutting down
[DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0
[DEBUG/MainProcess] running the remaining "atexit" finalizers
"Success" (really silent failure, only able to replicate with Python 3.6):
[DEBUG/MainProcess] created semlock with handle 139710276231168
[DEBUG/MainProcess] created semlock with handle 139710276227072
[DEBUG/MainProcess] created semlock with handle 139710276222976
[DEBUG/MainProcess] Queue._after_fork()
[DEBUG/Process-1] Queue._after_fork()
[INFO/Process-1] child process calling self.run()
[DEBUG/Process-1] Queue._start_thread()
[DEBUG/Process-1] doing self._thread.start()
[DEBUG/Process-1] starting thread to feed data to pipe
[DEBUG/Process-1] ... done self._thread.start()
[DEBUG/Process-1] telling queue thread to quit
[INFO/Process-1] process shutting down
[INFO/Process-1] error in queue thread: [Errno 32] Broken pipe
[DEBUG/Process-1] running all "atexit" finalizers with priority >= 0
[DEBUG/Process-1] running the remaining "atexit" finalizers
[DEBUG/Process-1] joining queue thread
[DEBUG/Process-1] ... queue thread joined
[INFO/Process-1] process exiting with exitcode 0
[INFO/MainProcess] process shutting down
[DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0
[DEBUG/MainProcess] running the remaining "atexit" finalizers
True success (using either time.sleep(0.01)):
[DEBUG/MainProcess] created semlock with handle 140283921616896
[DEBUG/MainProcess] created semlock with handle 140283921612800
[DEBUG/MainProcess] created semlock with handle 140283921608704
[DEBUG/MainProcess] Queue._after_fork()
[DEBUG/Process-1] Queue._after_fork()
[INFO/Process-1] child process calling self.run()
[DEBUG/Process-1] Queue._start_thread()
[DEBUG/Process-1] doing self._thread.start()
[DEBUG/Process-1] starting thread to feed data to pipe
[DEBUG/Process-1] ... done self._thread.start()
[DEBUG/Process-1] telling queue thread to quit
[INFO/Process-1] process shutting down
[DEBUG/Process-1] feeder thread got sentinel -- exiting
[DEBUG/Process-1] running all "atexit" finalizers with priority >= 0
[DEBUG/Process-1] running the remaining "atexit" finalizers
[DEBUG/Process-1] joining queue thread
[DEBUG/Process-1] ... queue thread joined
[INFO/Process-1] process exiting with exitcode 0
[INFO/MainProcess] process shutting down
[DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0
[DEBUG/MainProcess] running the remaining "atexit" finalizers
The difference seems to be that in the truly successful case, the feeder receives the sentinel object before the atexit handlers.

the primary issue with your code is that nobody is consuming what your worker process has put in the queue. python queues expect that the data in queues is consumed ("flushed to pipe") prior to the process that put data on it is killed.
in this light, your example doesn't make much sense, but if you want to get it to work:
the key is the queue.cancel_join_thread() -- https://docs.python.org/3/library/multiprocessing.html
Warning As mentioned above, if a child process has put items on a
queue (and it has not used JoinableQueue.cancel_join_thread), then
that process will not terminate until all buffered items have been
flushed to the pipe. This means that if you try joining that process
you may get a deadlock unless you are sure that all items which have
been put on the queue have been consumed. Similarly, if the child
process is non-daemonic then the parent process may hang on exit when
it tries to join all its non-daemonic children.
Note that a queue created using a manager does not have this issue
^ relevant bit. the issue is that stuff is being put on the queue from the child process but NOT consumed by anyone. In this case cancel_join_queue must be called on the CHILD process prior to asking it to join. This code sample will get rid of the error.
import multiprocessing
import time
import logging
import multiprocessing.util
multiprocessing.util.log_to_stderr(level=logging.DEBUG)
queue = multiprocessing.Queue(maxsize=10)
def worker(queue):
queue.put('abcdefghijklmnop')
# "Indicate that no more data will be put on this queue by the
# current process." --Documentation
# time.sleep(0.01)
queue.close()
queue.cancel_join_thread() # ideally, this would not be here but would rather be a response to a signal (or other IPC message) sent from the main process
proc = multiprocessing.Process(target=worker, args=(queue,))
proc.start()
# "Indicate that no more data will be put on this queue by the current
# process." --Documentation
# time.sleep(0.01)
queue.close()
proc.join()
I didn't bother with IPC for this because there's no consumer at all but I hope the idea is clear.

Python multiprocessing daemon vs non-daemon vs main

I'm learning multiprocessing in Python while I found this odd behaviour between daemon and non-daemon process with respect to the main process.
My Code:
import multiprocessing
import time
def worker(name,num):
print name, 'Starting'
time.sleep(num)
print name, 'Exiting'
if __name__ == '__main__':
print 'main starting'
p1=multiprocessing.Process(target=worker, args=("p1",7,))
p2=multiprocessing.Process(target=worker, args=("p2",5,))
p2.daemon=True
p1.start()
p2.start()
time.sleep(4)
print 'main exiting'
The output I'm getting is:
main starting
p1 Starting
p2 Starting
main exiting
p1 Exiting
Expected Output:
main starting
p1 Starting
p2 Starting
main exiting
p2 Exiting
p1 Exiting
After few searches, I found this answer and inserted the following line to my code.
logger = multiprocessing.log_to_stderr(logging.INFO)
And the output I got is,
main starting
[INFO/p1] child process calling self.run()
p1 Starting
[INFO/p2] child process calling self.run()
p2 Starting
main exiting
[INFO/MainProcess] process shutting down
[INFO/MainProcess] calling terminate() for daemon p2
[INFO/MainProcess] calling join() for process p2
[INFO/MainProcess] calling join() for process p1
p1 Exiting
[INFO/p1] process shutting down
[INFO/p1] process exiting with exitcode 0
As per my understanding,
When a main process exits, it terminates all of its child daemon processes. (Which can be seen here)
A main process can not exit before all of its child non-daemon processes exit.
But here, why the main process is trying to shut down before p1 exits?
p1 starts and sleeps for 7 seconds.
p2 starts and sleeps for 5 seconds.
main process sleeps for 4 seconds.
main process wakes up after 4 seconds and waits for p1 to exit.
p2 wakes up after 5 seconds and exits.
p1 wakes up after 7 seconds and exits.
main process exits.
Wouldn't be the above a normal time-line for the above program?
Can anyone please explain what's happening here and why?
EDIT
After adding the line p1.join() at the end of the code, I'm getting the following output:
main starting
[INFO/Process-1] child process calling self.run()
p1 Starting
[INFO/Process-2] child process calling self.run()
p2 Starting
main exiting
p2 Exiting
[INFO/Process-2] process shutting down
[INFO/Process-2] process exiting with exitcode 0
p1 Exiting
[INFO/Process-1] process shutting down
[INFO/Process-1] process exiting with exitcode 0
[INFO/MainProcess] process shutting down

When you see
[INFO/MainProcess] calling join() for process p1
it means the main process has not exited yet -- it's in the process of shutting down, but of course will not be done shutting down until that join returns... which will happen only once the joined process is done.
So the timeline is indeed as you expect it to be -- but since that join to p1 is the very last think the main process ever does, you don't see that in the output or logging from it. (main has taken all the termination-triggered action, but as a process it's still alive until then).
To verify, use (on a Unixy system) ps from another terminal while running this (perhaps with slightly longer delays to help you check): you will never see a single Python process running out of this complex -- there will be two (main and p1) until the end.

Python Multiprocess| Daemon & Join
Question end up adding more wait towards the behavior of daemon flag and join method, hence here a quick explanation with a simple script.
The daemon process is terminated automatically before the main program exits, to avoid leaving orphaned processes running but is not terminated with the main program. So any outstanding processes left by the daemoned function won't run!
However if the join() method is called on the daemoned function then the main programm will wait the left over processes.
INPUT
import multiprocessing
import time
import logging
def daemon():
p = multiprocessing.current_process()
print('Starting:', p.name, p.pid, flush=True)
print('---' * 15)
time.sleep(2)
print('===' * 15 + ' < ' + f'Exiting={p.name, p.pid}' + ' > ' + '===' * 15, flush=True)
def non_daemon():
p = multiprocessing.current_process()
print('Starting:', p.name, p.pid, flush=True)
print('---'*15)
print('===' * 15 + ' < ' + f'Exiting={p.name, p.pid}' + ' > ' + '===' * 15, flush=True)
if __name__ == '__main__':
print('==='*15 + ' < ' + 'MAIN PROCESS' + ' > ' + '==='*15)
logger = multiprocessing.log_to_stderr(logging.INFO)
# DAEMON
daemon_proc = multiprocessing.Process(name='MyDaemon', target=daemon)
daemon_proc.daemon = True
# NO-DAEMON
normal_proc = multiprocessing.Process(name='non-daemon', target=non_daemon)
normal_proc.daemon = False
daemon_proc.start()
time.sleep(2)
normal_proc.start()
# daemon_proc.join()
OUTPUT NO JOIN METHOD##
============================================= < MAIN PROCESS > =============================================
Starting: MyDaemon 8448
---------------------------------------------
[INFO/MyDaemon] child process calling self.run()
[INFO/MainProcess] process shutting down
[INFO/MainProcess] calling terminate() for daemon MyDaemon
[INFO/MainProcess] calling join() for process MyDaemon
[INFO/MainProcess] calling join() for process non-daemon
Starting: non-daemon 6688
---------------------------------------------
============================================= < Exiting=('non-daemon', 6688) > =============================================
[INFO/non-daemon] child process calling self.run()
[INFO/non-daemon] process shutting down
[INFO/non-daemon] process exiting with exitcode 0
Process finished with exit code 0
The function daemon() is 'spleeping' 2 second, hence as Python reaches the bottom of the script the program shut down but not daemon().
Note:
[INFO/MainProcess] calling terminate() for daemon MyDaemon
Now if the last line of the script daemon_proc.join() is not commented.
OUTPUT + JOIN METHOD
============================================= < MAIN PROCESS > =============================================
Starting: MyDaemon 13588
---------------------------------------------
[INFO/MyDaemon] child process calling self.run()
============================================= < Exiting=('MyDaemon', 13588) > =============================================
[INFO/MyDaemon] process shutting down
[INFO/MyDaemon] process exiting with exitcode 0
[INFO/MainProcess] process shutting down
[INFO/MainProcess] calling join() for process non-daemon
Starting: non-daemon 13608
---------------------------------------------
============================================= < Exiting=('non-daemon', 13608) > =============================================
[INFO/non-daemon] child process calling self.run()
[INFO/non-daemon] process shutting down
[INFO/non-daemon] process exiting with exitcode 0
Process finished with exit code 0