PyAudio and Matplotlib crash when used together - python

Running the following code:
import threading
import pyaudio
from matplotlib import pyplot as plt
def output():
p = pyaudio.PyAudio()
stream_ = p.open(format=pyaudio.paFloat32,
channels=1,
rate=8000,
output=True)
stream_.stop_stream()
stream_.close()
p.terminate()
output_thread = threading.Thread(target=output, args=())
output_thread.start()
output_thread.join()
fig = plt.figure()
ax0 = fig.add_subplot(111)
ax0.plot([1,2,3])
plt.show()
causes Python to crash with the error below. How might I solve this? I am running Python 3.8, PyAudio 0.2.11 and Matplotlib 3.3.1 and Mac Os version 10.15.5.
Crashed Thread: 0 Dispatch queue: com.apple.main-thread
Exception Type: EXC_BAD_INSTRUCTION (SIGILL)
Exception Codes: 0x0000000000000001, 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY
Termination Signal: Illegal instruction: 4
Termination Reason: Namespace SIGNAL, Code 0x4
Terminating Process: exc handler [4840]
Application Specific Information:
The current event queue and the main event queue are not the same. This is probably because _TSGetMainThread was called for the first time off the main thread. _TSGetMainThread was called for the first time here:
0 CarbonCore 0x00007fff36993345 _TSGetMainThread + 138
1 CarbonCore 0x00007fff3699324a GetThreadGlobals + 26
2 CarbonCore 0x00007fff3699d5e4 NewPtrClear + 14
3 CarbonCore 0x00007fff369b4aba AVLInit + 62
4 CarbonCore 0x00007fff369b49f1 __INIT_Folders_block_invoke + 9
5 libdispatch.dylib 0x00007fff6f603658 _dispatch_client_callout + 8
6 libdispatch.dylib 0x00007fff6f6047de _dispatch_once_callout + 20

Ok if I run:
fig = plt.figure()
before I create the thread I can avoid this crash. I'm guessing this allows me to call _TSGetMainThread on the main thread for the first time.

Related

setting rpi-gpio pin on high after running a python script

I'm building a photogrametry setup with raspberry pi and stepper motor.
The python script runs fine, but i got a problem with setting a pin to high after the script ran through.
The stepper driver has an enable input, which diasables the motor with high input, so i set the pin (gpio26) on high on boot with pigpio, this works fine. While running the python script, the pin is set on low, so the stepper can proceed, after proceeding i want to set the pin on high again.
i tried following commands:
os.system('pigs w 26 1') and
subprocess.call("pigs w 26 1", shell=True)
for a moment they work, but after exiting the script the pin is set on low again.
It's like the commands are resetted after the script stops.
Where is my fault?
Thank you
Edit:
Here is the gpio related code:
import os, sys
import subprocess
from time import sleep
from gpiozero import DigitalOutputDevice as stepper
def InitGPIO():
try:
global step_pul #pulse
global step_en #enable
step_pul=stepper(21)
step_en=stepper(26)
print ("GPIO initialisiert.")
except:
print ("Fehler bei der GPIO-Initialisierung.")
def motor_step():
SPR=40000 #steps per rotation
step_count = SPR
delay = .000025
for x in range(step_count):
step_pul.on()
sleep(delay)
step_pul.off()
sleep(delay)
InitGPIO()
step_en.off()
for i in range(1):
#camTrigger(1)
motor_step()
#os.system('sudo -u root -S pigs w 26 1')
subprocess.call("pigs w 26 1", shell=True)
When i type pigs w 26 1 in the shell, it works...
To make my comment an answer:
Gpiozero only resets the pins it touches, so if you don't initialize or touch pin 26 with gpiozero (i.e. replace step_en.off() with pigs w 26 0 and don't even initialize step_en), it shouldn't also reset the pin:
import os
import time
import gpiozero
step_pul = gpiozero.DigitalOutputDevice(21)
def motor_step():
SPR = 40000 # steps per rotation
step_count = SPR
delay = .000025
for x in range(step_count):
step_pul.on()
time.sleep(delay)
step_pul.off()
time.sleep(delay)
# Enable motor
os.system("pigs w 26 0")
for i in range(1):
# camTrigger(1)
motor_step()
# Disable motor
os.system("pigs w 26 1")

Kivy application unexpectedly halting when using Threads

I am not sure whether this is an iOS issue or whether this is an issue with Kivy or even with Python (e.g. https://bugs.python.org/issue37788), but I am experiencing some problems with threading.
I have built an iPad app using the Kivy framework that makes several calls to an API, and uses the threading module to asynchronously make requests. Below is the code that handles the API requests:
import json
import requests
import base64
import threading
def thread(function):
def wrap(*args, **kwargs):
t = threading.Thread(target=function, args=args, kwargs=kwargs)
t.start()
return t
return wrap
class MathPixAPI:
stroke_url = '*******************'
header = {
"content-type": "application/json",
"app_id": "*******************",
"app_key": "*******************"
}
#thread
def post_data(self, file_name: str, root):
"""
Posts a base64 encoded image to the MathPixAPI then updates the data DictProperty of the ExpressionWriter that
calls this function
:param file_name: The name of the file - e.g. "image.png"
:param root: The ExpressionWriter that calls the function
"""
image_uri = "data:image/png;base64," + base64.b64encode(open(file_name, "rb").read()).decode()
r = requests.post("https://api.mathpix.com/v3/text",
data=json.dumps({'src': image_uri}),
headers=self.header)
root.data = json.loads(r.text)
The app makes no more than 5 asynchronous requests at one time, and is called from the function below:
def get_image_data(self):
"""
The function first saves the ExpressionWriter.canvas as a PNG file to the user_data_directory (automatically
determined depending on the device the user is running the app on). Then this images is sent to the MathPix API
which then return data on the handwritten answer (see api.py for more details). The api call updates self.data
which in turn calls self._on_data().
"""
file_name = f'{App.get_running_app().user_data_dir}/image_{self.number}.png'
self.export_to_png(file_name)
MathPixAPI().post_data(file_name, self)
This works really well, up until the 20th-25th request, upon which the program halts. In Xcode I receive the following error log:
021-04-09 18:11:02.300179+0100 ccc-writer-3[4261:4790641] [Animation] +[UIView setAnimationsEnabled:] being called from a background thread. Performing any operation from a background thread on UIView or a subclass is not supported and may result in unexpected and insidious behavior. trace=(
0 UIKitCore 0x0000000187cbb538 8518EAE3-832B-3FF0-9FA5-9DBE3041F26C + 17859896
1 libdispatch.dylib 0x0000000101ce56c0 _dispatch_client_callout + 20
2 libdispatch.dylib 0x0000000101ce71f8 _dispatch_once_callout + 136
3 UIKitCore 0x0000000187cbb4bc 8518EAE3-832B-3FF0-9FA5-9DBE3041F26C + 17859772
4 UIKitCore 0x0000000187cbb628 8518EAE3-832B-3FF0-9FA5-9DBE3041F26C + 17860136
5 UIKitCore 0x0000000187abbd64 8518EAE3-832B-3FF0-9FA5-9DBE3041F26C + 15764836
6 UIKitCore 0x0000000187aae150 8518EAE3-832B-3FF0-9FA5-9DBE3041F26C + 15708496
7 UIKitCore 0x00000001877b2f20 8518EAE3-832B-3FF0-9FA5-9DBE3041F26C + 12582688
8 UIKitCore 0x0000000187cb2b30 8518EAE3-832B-3FF0-9FA5-9DBE3041F26C + 17824560
9 UIKitCore 0x0000000187aacd50 8518EAE3-832B-3FF0-9FA5-9DBE3041F26C + 15703376
10 ccc-writer-3 0x0000000100822960 -[SDL_uikitviewcontroller showKeyboard] + 108
11 ccc-writer-3 0x0000000100823164 UIKit_ShowScreenKeyboard + 60
12 ccc-writer-3 0x00000001007ec490 SDL_StartTextInput + 92
... [A whole bunch of memory addresses] ...
74 ccc-writer-3 0x0000000100610df4 _PyEval_EvalFrameDefault + 5432
75 ccc-writer-3 0x000000010054dfe0 function_code_fastcall + 120
76 ccc-writer-3 0x00000001005505f8 method_vectorcall + 264
77 ccc-writer-3 0x000000010054d95c PyVectorcall_Call + 104
78 ccc-writer-3 0x0000000100770c40 t_bootstrap + 80
79 ccc-writer-3 0x000000010065e8e8 pythread_wrapper + 28
80 libsystem_pthread.dylib 0x00000001cfbb3cb0 _pthread_start + 320
81 libsystem_pthread.dylib 0x00000001cfbbc778 thread_start + 8
)
2021-04-09 18:11:02.308745+0100 ccc-writer-3[4261:4790641] *** Assertion failure in -[_UISimpleFenceProvider trackSystemAnimationFence:], _UISimpleFenceProvider.m:51
2021-04-09 18:11:02.311976+0100 ccc-writer-3[4261:4790641] *** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'main thread only'
*** First throw call stack:
(0x184dc686c 0x199de1c50 0x184ccc000 0x18606091c 0x186cd20bc 0x187777d30 0x1877cb888 0x186c00e58 0x1875b2610 0x1871c71b8 0x1871c54d0 0x1871c51f0 0x1871c674c 0x1871c67c8 0x1871c682c 0x187541c94 0x1871c3478 0x1871c2b88 0x1877b7f58 0x1877b2fc8 0x187cb2b30 0x187aacd50 0x100822960 0x100823164 0x1007ec490 0x1008d30b8 0x10055680c 0x100614e6c 0x100610df4 0x100615e98 0x10054e160 0x10055058c 0x100614e6c 0x100611d50 0x10054dfe0 0x100614e6c 0x100610df4 0x10054dfe0 0x100614e6c 0x100610df4 0x100615e98 0x10054e160 0x100550664 0x10054d95c 0x100612888 0x100615e98 0x10060f87c 0x100859f50 0x10085e5dc 0x10085f020 0x100bc7598 0x100bc5c78 0x100beac94 0x100591568 0x1005908dc 0x100c116a0 0x1005908dc 0x100610d94 0x10054dfe0 0x100614e6c 0x100610df4 0x100615e98 0x10054e160 0x10055058c 0x100614e6c 0x100611d50 0x100615e98 0x10060f87c 0x100859f50 0x10085e5dc 0x10085f020 0x100bc7598 0x100bc5c78 0x100bcbdc8 0x100beac94 0x100591568 0x1005908dc 0x100610d94 0x10054dfe0 0x10054d95c 0x100612888 0x10054dfe0 0x100614e6c 0x100610df4 0x10054dfe0 0x100614e6c 0x100610df4 0x10054dfe0 0x1005505f8 0x10054d95c 0x100770c40 0x10065e8e8 0x1cfbb3cb0 0x1cfbbc778)
libc++abi.dylib: terminating with uncaught exception of type NSException
*** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'main thread only'
terminating with uncaught exception of type NSException
I am not sure what "main thread only" means nor do I have any idea how to resolve this issue. Can anyone clarify what this means and explain what the problem is? How can I stop my program from halting?
Thanks in advance.
I fixed this issue in Xcode by amending the runtime API checking.
Navigate to:
Product > Scheme > Edit Scheme > Run / Debug > Diagnostics
then deselect Main Thread Checker

Python is crashing due to libdispatch crashing child thread

I am using the pynetdicom library to receive and process medical dicom images. The processing is performed in the callback function "on_association_released". However, when receiving certain studies, it will cause Python to crash due to what appears to be a child thread crashing.
From the OSX crash report it seems to be libdispatch library that is the cause but not sure how or why.
This is the function:
def on_association_released(self):
if not self.auto_process:
self.incoming = []
return
dicoms = [Dicom(f=x) for x in self.incoming]
self.incoming = []
incoming = Study(dicom_list=dicoms)
log.info("Incoming study: {incoming}".format(**locals()))
completed_tasks = {}
time.sleep(1)
for task in AVAILABLE_PROCESS_TASKS:
log.info("Trying task: {task}".format(**locals()))
process_task = task(study=incoming)
try:
if process_task.valid:
log.info("{incoming} is valid for {process_task}".format(**locals()))
try:
process_task.process()
except Exception as e:
log.warning(
'Failed to perform {process_task} on {incoming}: \n {e}'.format(**locals())
)
else:
log.info("Completed {process_task} for {incoming} !".format(**locals()))
else:
log.warning("{incoming} is not a valid study for {process_task}".format(**locals()))
except Exception as e:
log.warning("{incoming} could not be assessed by {process_task}".format(**locals()))
myemail.nhs_mail(recipients=[admin],
subject=f"dicomserver {VERSION}: Failed to start listener",
message=f"{incoming} could not be assessed by {process_task}: {e.args}"
)
This is the final log message from the application log:
2019-03-15 12:19:06 I [process.py:on_association_released:171] Incoming study: Study(1.2.826.0.1.2112370.55.1.12145941)
This is the OSX Crash Report:
Process: Python [84177]
Path: /Library/Frameworks/Python.framework/Versions/3.6/Resources/Python.app/Contents/MacOS/Python
Identifier: Python
Version: 3.6.1 (3.6.1)
Code Type: X86-64 (Native)
Parent Process: Python [84175]
Responsible: Terminal [346]
User ID: 503
Date/Time: 2019-03-15 12:19:06.371 +0000
OS Version: Mac OS X 10.11.6 (15G1108)
Report Version: 11
Anonymous UUID: E7340644-9523-1C6B-0B2B-74D6043CFED6
Time Awake Since Boot: 590000 seconds
System Integrity Protection: enabled
Crashed Thread: 1
Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000110
VM Regions Near 0x110:
-->
__TEXT 0000000100000000-0000000100001000 [ 4K] r-x/rwx SM=COW /Library/Frameworks/Python.framework/Versions/3.6/Resources/Python.app/Contents/MacOS/Python
Application Specific Information:
*** multi-threaded process forked ***
crashed on child side of fork pre-exec
This is the top of Thread 1 crash trace:
Thread 1 Crashed:
0 libdispatch.dylib 0x00007fff8e6cc661 _dispatch_queue_push_queue + 345
1 libdispatch.dylib 0x00007fff8e6cab06 _dispatch_queue_wakeup_with_qos_slow + 126
2 libdispatch.dylib 0x00007fff8e6d113f _dispatch_mach_msg_send + 1952
3 libdispatch.dylib 0x00007fff8e6d08dc dispatch_mach_send + 262
4 libxpc.dylib 0x00007fff86858fc9 xpc_connection_send_message_with_reply + 131
5 com.apple.CoreFoundation 0x00007fff8ef43b3f __66-[CFPrefsSearchListSource generationCountFromListOfSources:count:]_block_invoke_2 + 143
6 com.apple.CoreFoundation 0x00007fff8ef4396d _CFPrefsWithDaemonConnection + 381
7 com.apple.CoreFoundation 0x00007fff8ef42af6 __66-[CFPrefsSearchListSource generationCountFromListOfSources:count:]_block_invoke + 150
8 com.apple.CoreFoundation 0x00007fff8ef42893 -[CFPrefsSearchListSource generationCountFromListOfSources:count:] + 179
9 com.apple.CoreFoundation 0x00007fff8ef42174 -[CFPrefsSearchListSource alreadylocked_copyDictionary] + 324
10 com.apple.CoreFoundation 0x00007fff8ef41dbc -[CFPrefsSearchListSource alreadylocked_copyValueForKey:] + 60
11 com.apple.CoreFoundation 0x00007fff8ef41d4c ___CFPreferencesCopyAppValueWithContainer_block_invoke + 60
12 com.apple.CoreFoundation 0x00007fff8ef39a70 +[CFPrefsSearchListSource withSearchListForIdentifier:container:perform:] + 608
13 com.apple.CoreFoundation 0x00007fff8ef397c7 _CFPreferencesCopyAppValueWithContainer + 183
14 com.apple.SystemConfiguration 0x00007fff998b3a9b SCDynamicStoreCopyProxiesWithOptions + 163
15 _scproxy.cpython-36m-darwin.so 0x000000010f0f5a63 get_proxy_settings + 35
16 org.python.python 0x000000010006a604 _PyCFunction_FastCallDict + 436
17 org.python.python 0x00000001000f33e4 call_function + 612
18 org.python.python 0x00000001000f8d84 _PyEval_EvalFrameDefault + 21892
The issue looks awfully similar to a long-standing problem with Python on MacOS.
The root cause as far as I understand it is that fork() is hard to do right if there's threads involved, unless you immediately exec().
MacOS "protects" again the possible pitfalls by crashing a process if it's accessing certain system functionality such as libdispatch if it forked, but didn't exec yet.
Unfortunately these calls can happen in unexpected places, such as in _scproxy.cpython-36m-darwin.so which is shown at position 15 of the stack trace.
There's a number of Python bugs filed about this (1, 2, 3, for example), but there's no silver bullet as far as I know.
In your particular case, it might be possible to prevent the crash by running your Python interpreter with the environment variable no_proxy=*. This should prevent calls to the system configuration framework scproxy to find proxy settings.

Combine using Python multiprocessing ThreadPool + Pool (for process), process does not work sometimes

I'm using ThreadPool for IO-intensive work (such as loading data) while fork a process within the thread if there is CPU-intensive calculation needed. This is my workaround to the GIL problem (anyway, this is not my key problem here).
My problem is: when running my code, sometimes, although a process is folked, it looks always in sleeping status (not even run my calculation). Thus, it causes blocking since the code calls join to wait for result (or error). Note that the problem doesn't always happen, just occasionally.
My running environment is: linux centos 7.3, anaconda 5.1.0 (with built-in python 3.6.4). And note that: i failed to reproduce the issue with the same code on windows.
the following is the simplified code, which can reproduce the issue on linux:
import logging
import time
import random
import os
import threading
from multiprocessing.pool import Pool, ThreadPool
class Task(object):
def __init__(self, func, *args) -> None:
super().__init__()
self.func = func
self.args = args
class ConcurrentExecutor(object):
def __init__(self, spawn_count=1) -> None:
super().__init__()
self.spawn_count = spawn_count
def fork(self, task):
result = None
print('Folk started: %s' % task.args)
pool = Pool(processes=1)
try:
handle = pool.apply_async(task.func, task.args)
pool.close()
result = handle.get()
print('Folk completed: %s' % task.args)
except Exception as err:
print('Fork failure: FOLK%s' % task.args)
raise err
finally:
pool.terminate()
return result
def spawn(self, tasks):
results = []
try:
print('Spawn started')
handles = []
pool = ThreadPool(processes=self.spawn_count)
for task in tasks:
handles.append(pool.apply_async(task.func, task.args))
pool.close()
pool.join()
print('all done')
for handle in handles:
results.append(handle.get(10))
print('Spawn completed')
except Exception as err:
print('Spawn failure')
raise err
return results
def foo_proc(i):
print(i)
result=i*i
time.sleep(1)
return result
def foo(i):
executor = ConcurrentExecutor(2)
try:
result = executor.fork(Task(foo_proc, i))
except Exception as err:
result = 'ERROR'
return result
if __name__ == '__main__':
executor = ConcurrentExecutor(4)
tasks = []
for i in range(1, 10):
tasks.append(Task(foo, i))
start = time.time()
print(executor.spawn(tasks))
end = time.time()
print(end - start)
The following shows an example of running result:
[appadmin#168-61-40-47 test]$ python test.py
7312
Spawn started
Folk started: 1
Folk started: 2
Folk started: 3
Folk started: 4
4
2
1
3
Folk completed: 4
Folk completed: 2
Folk completed: 1
Folk completed: 3
Folk started: 5
Folk started: 6
Folk started: 7
5
Folk started: 8
7
8
Folk completed: 5
Folk completed: 7
Folk completed: 8
Folk started: 9
9
Folk completed: 9
You may see the code was stuck since process "6" was folked but never do the work. Meanwhile, i can see two python processes, which is supposed to be only 1 process at last if everything runs correctly:
[user#x-x-x-x x]$ ps -aux | grep python
user 7312 0.1 0.0 1537216 13524 pts/0 Sl+ 22:23 0:02 python test.py
user 7339 0.0 0.0 1545444 10604 pts/0 S+ 22:23 0:00 python test.py
Could anyone help? Thanks in advance!

SIGILL after fork in Sage/Python

I'm doing some calculations with Sage.
I am playing around with fork. I have a very simple test case which is basically like this:
def fork_test():
import os
pid = os.fork()
if pid != 0:
print "parent, child: %i" % pid
os.waitpid(pid, 0)
else:
print "child"
try:
# some dummy matrix calculation
finally:
os._exit(0)
(Look below for _fork_test_func() for some matrix calculations.)
And I'm getting:
------------------------------------------------------------------------
Unhandled SIGILL: An illegal instruction occurred in Sage.
This probably occurred because a *compiled* component of Sage has a bug
in it and is not properly wrapped with sig_on(), sig_off(). You might
want to run Sage under gdb with 'sage -gdb' to debug this.
Sage will now terminate.
------------------------------------------------------------------------
With this (incomplete) backtrace:
Crashed Thread: 0 Dispatch queue: com.apple.root.default-priority
Exception Type: EXC_BAD_INSTRUCTION (SIGILL)
Exception Codes: 0x0000000000000001, 0x0000000000000000
Application Specific Information:
BUG IN LIBDISPATCH: flawed group/semaphore logic
Thread 0 Crashed:: Dispatch queue: com.apple.root.default-priority
0 libsystem_kernel.dylib 0x00007fff8c6d1d46 __kill + 10
1 libcsage.dylib 0x0000000101717f33 sigdie + 124
2 libcsage.dylib 0x0000000101717719 sage_signal_handler + 364
3 libsystem_c.dylib 0x00007fff86b1094a _sigtramp + 26
4 libdispatch.dylib 0x00007fff89a66c74 _dispatch_thread_semaphore_signal + 27
5 libdispatch.dylib 0x00007fff89a66f3e _dispatch_apply2 + 143
6 libdispatch.dylib 0x00007fff89a66e30 dispatch_apply_f + 440
7 libBLAS.dylib 0x00007fff906ca435 APL_dtrsm + 1963
8 libBLAS.dylib 0x00007fff906702b6 cblas_dtrsm + 882
9 matrix_modn_dense_double.so 0x0000000108612615 void FFLAS::Protected::ftrsmRightLowerNoTransUnit<double>::delayed<FFPACK::Modular<double> >(FFPACK::Modular<double> const&, unsigned long, unsigned long, FFPACK::Modular<double>::Element*, unsigned long, FFPACK::Modular<double>::Element*, unsigned long, unsigned long, unsigned long) + 2853
10 matrix_modn_dense_double.so 0x0000000108611daa void FFLAS::Protected::ftrsmRightLowerNoTransUnit<double>::delayed<FFPACK::Modular<double> >(FFPACK::Modular<double> const&, unsigned long, unsigned long, FFPACK::Modular<double>::Element*, unsigned long, FFPACK::Modular<double>::Element*, unsigned long, unsigned long, unsigned long) + 698
11 matrix_modn_dense_double.so 0x0000000108612ccf void FFLAS::Protected::ftrsmRightLowerNoTransUnit<double>::operator()<FFPACK::Modular<double> >(FFPACK::Modular<double> const&, unsigned long, unsigned long, FFPACK::Modular<double>::Element*, unsigned long, FFPACK::Modular<double>::Element*, unsigned long) + 831
12 ??? 0x00007f99e481a028 0 + 140298940424232
Thread 1:
0 libsystem_kernel.dylib 0x00007fff8c6d26d6 __workq_kernreturn + 10
1 libsystem_c.dylib 0x00007fff86b24f4c _pthread_workq_return + 25
2 libsystem_c.dylib 0x00007fff86b24d13 _pthread_wqthread + 412
3 libsystem_c.dylib 0x00007fff86b0f1d1 start_wqthread + 13
Thread 2:
0 libsystem_kernel.dylib 0x00007fff8c6d26d6 __workq_kernreturn + 10
1 libsystem_c.dylib 0x00007fff86b24f4c _pthread_workq_return + 25
2 libsystem_c.dylib 0x00007fff86b24d13 _pthread_wqthread + 412
3 libsystem_c.dylib 0x00007fff86b0f1d1 start_wqthread + 13
Thread 0 crashed with X86 Thread State (64-bit):
rax: 0x0000000000000000 rbx: 0x00007fff5ec8e418 rcx: 0x00007fff5ec8df28 rdx: 0x0000000000000000
rdi: 0x000000000000b8f7 rsi: 0x0000000000000004 rbp: 0x00007fff5ec8df40 rsp: 0x00007fff5ec8df28
r8: 0x00007fff5ec8e418 r9: 0x0000000000000000 r10: 0x000000000000000a r11: 0x0000000000000202
r12: 0x00007f99ea500de0 r13: 0x0000000000000003 r14: 0x00007fff5ec8e860 r15: 0x00007fff906ca447
rip: 0x00007fff8c6d1d46 rfl: 0x0000000000000202 cr2: 0x00007fff74a29848
Logical CPU: 0
Is there something special I need to do after a fork? I looked up the fork decorator of Sage and it looks like it basically does the same.
The crash also happens with the fork decorator of Sage itself. Another test case:
def fork_test2():
def test():
# do some stuff
from sage.parallel.decorate import fork
test_ = fork(test, verbose=True)
test_()
Even simpler test case:
def _fork_test_func():
while True:
m = matrix(QQ, 100, [randrange(-100,100) for i in range(100*100)])
m.right_kernel()
def fork_test():
import os
pid = os.fork()
if pid != 0:
print "parent, child: %i" % pid
os.waitpid(pid, 0)
else:
print "child"
try:
_fork_test_func()
finally:
os._exit(0)
Results in a slightly different crash:
python(48672) malloc: *** error for object 0x11185f000: pointer being freed already on death-row
*** set a breakpoint in malloc_error_break to debug
With backtrace:
Crashed Thread: 1 Dispatch queue: com.apple.root.default-priority
Exception Type: EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Application Specific Information:
*** error for object 0x11185f000: pointer being freed already on death-row
Thread 0:: Dispatch queue: com.apple.main-thread
0 matrix2.so 0x0000000107fa403f __pyx_pw_4sage_6matrix_7matrix2_6Matrix_71right_kernel_matrix + 27551
1 ??? 0x000000000000000d 0 + 13
Thread 1 Crashed:: Dispatch queue: com.apple.root.default-priority
0 libsystem_kernel.dylib 0x00007fff8c6d239a __semwait_signal_nocancel + 10
1 libsystem_c.dylib 0x00007fff86b17e1b nanosleep$NOCANCEL + 138
2 libsystem_c.dylib 0x00007fff86b7b9a8 usleep$NOCANCEL + 54
3 libsystem_c.dylib 0x00007fff86b67eca __abort + 203
4 libsystem_c.dylib 0x00007fff86b67dff abort + 192
5 libsystem_c.dylib 0x00007fff86b43905 szone_error + 580
6 libsystem_c.dylib 0x00007fff86b43f7d free_large + 229
7 libsystem_c.dylib 0x00007fff86b3b8f8 free + 199
8 libBLAS.dylib 0x00007fff906b0431 __APL_dgemm_block_invoke_0 + 132
9 libdispatch.dylib 0x00007fff89a65f01 _dispatch_call_block_and_release + 15
10 libdispatch.dylib 0x00007fff89a620b6 _dispatch_client_callout + 8
11 libdispatch.dylib 0x00007fff89a631fa _dispatch_worker_thread2 + 304
12 libsystem_c.dylib 0x00007fff86b24d0b _pthread_wqthread + 404
13 libsystem_c.dylib 0x00007fff86b0f1d1 start_wqthread + 13
The same happens also for this:
def fork_test2():
from sage.parallel.decorate import fork
test_ = fork(_fork_test_func, verbose=True)
test_()
-- but only if you used some other matrix calculations before.
This test case also works on a fresh Sage session:
def _fork_test_func(iterator=None):
if not iterator:
import itertools
iterator = itertools.count()
for i in iterator:
m = matrix(QQ, 100, [randrange(-100,100) for i in range(100*100)])
m.right_kernel()
def fork_test():
_fork_test_func(range(10))
import os
pid = os.fork()
if pid != 0:
print "parent, child: %i" % pid
os.waitpid(pid, 0)
else:
print "child"
try:
_fork_test_func()
finally:
os._exit(0)
I have downloaded the binaries for MacOSX 64bit of Sage 5.8.
(Note that I also asked on ask.sagemath.org here.)
Both of these crashreports indicate that a multi-threaded process fork()ed, which greatly restricts the set of operations that are safe to execute in the child, essentially you can only call execve() et al, along with a few other functions from the list of async-signal-safe functions
This is documented in the CAVEATS section of the fork(2) manpage as well as in the standard:
A process shall be created with a single thread. If a multi-threaded process calls fork(), the new process shall contain a replica of the calling thread and its entire address space, possibly including the states of mutexes and other resources. Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of the exec functions is called.
Since many APIs in Mac OS X frameworks will cause the process to become multithreaded, if you want the fork-child to be fully useable, you must limit you operations in the parent process before fork to APIs documented not to make a process multithreaded (essentially only POSIX APIs).

Categories