I'm running 64-bit Python 2.7.3 on Win7 64-bit. I can reliably crash the Python interpreter by doing this:
>>> from scipy import stats
>>> import time
>>> time.sleep(3)
and pressing Control-C during the sleep. A KeyboardInterrupt is not raised; the interpreter crashes. The following is printed:
forrtl: error (200): program aborting due to control-C event
Image PC Routine Line Source
libifcoremd.dll 00000000045031F8 Unknown Unknown Unknown
libifcoremd.dll 00000000044FC789 Unknown Unknown Unknown
libifcoremd.dll 00000000044E8583 Unknown Unknown Unknown
libifcoremd.dll 000000000445725D Unknown Unknown Unknown
libifcoremd.dll 00000000044672A6 Unknown Unknown Unknown
kernel32.dll 0000000077B74AF3 Unknown Unknown Unknown
kernel32.dll 0000000077B3F56D Unknown Unknown Unknown
ntdll.dll 0000000077C73281 Unknown Unknown Unknown
This makes it impossible to interrupt long-running scipy calculations.
Googling for "forrtl" and the like, I see suggestions that this kind of problem is due to use of a Fortran library that overrides Ctrl-C handling. I don't see a bug on the Scipy trackerbut given that Scipy is a library for use with Python, I would consider this a bug. It breaks Python's handling of Ctrl-C. Is there any workaround for this?
Edit: Following #cgohlke's suggestion I tried to add my own handler after importing scipy. This question about a related issue shows that adding a signal handler doesn't work. I tried using the Windows API SetConsoleCtrlHandler function via pywin32:
from scipy import stats
import win32api
def doSaneThing(sig, func=None):
print "Here I am"
raise KeyboardInterrupt
win32api.SetConsoleCtrlHandler(doSaneThing, 1)
After this, hitting Ctrl-C prints "Here I am", but Python still crashes with the forrtl error. Sometimes I also get a message saying "ConsoleCtrlHandler function failed", which quickly disappears.
If I run this in IPython, I can see a normal Python KeyboardInterrupt traceback before the forrtl error. I also see a normal Python traceback followed by the forrtl error if I raise some other error instead of KeyboardInterrupt (e.g., ValueError):
ValueError Traceback (most recent call last)
<ipython-input-1-08defde66fcb> in doSaneThing(sig, func)
3 def doSaneThing(sig, func=None):
4 print "Here I am"
----> 5 raise ValueError
6 win32api.SetConsoleCtrlHandler(doSaneThing, 1)
ValueError:
forrtl: error (200): program aborting due to control-C event
[etc.]
It seems that whatever the underlying handler is doing, it's not just trapping Ctrl-C directly, but is reacting to the error condition (ValueError) and crashing itself. Is there any way to eliminate this?
Here's a variation on your posted solution that may work. Maybe there's a better way to solve this problem -- or maybe even avoid it all together by setting an environment variable that tells the DLL to skip installing a handler. Hopefully this helps until you find a better way.
Both the time module (lines 868-876) and _multiprocessing module (lines 312-321) call SetConsoleCtrlHandler. In the case of the time module, its console control handler sets a Windows event, hInterruptEvent. For the main thread, time.sleep waits on this event via WaitForSingleObject(hInterruptEvent, ul_millis), where ul_millis is the number of milliseconds to sleep unless interrupted by Ctrl+C. Since the handler that you've installed returns True, the time module's handler never gets called to set hInterruptEvent, which means sleep cannot be interrupted.
I tried using imp.init_builtin('time') to reinitialize the time module, but apparently SetConsoleCtrlHandler ignores the 2nd call. It seems the handler has to be removed and then reinserted. Unfortunately, the time module doesn't export a function for that. So, as a kludge, just make sure you import the time module after you install your handler. Since importing scipy also imports time, you need to pre-load libifcoremd.dll using ctypes to get the handlers in the right order. Finally, add a call to thread.interrupt_main to make sure Python's SIGINT handler gets called[1].
For example:
import os
import imp
import ctypes
import thread
import win32api
# Load the DLL manually to ensure its handler gets
# set before our handler.
basepath = imp.find_module('numpy')[1]
ctypes.CDLL(os.path.join(basepath, 'core', 'libmmd.dll'))
ctypes.CDLL(os.path.join(basepath, 'core', 'libifcoremd.dll'))
# Now set our handler for CTRL_C_EVENT. Other control event
# types will chain to the next handler.
def handler(dwCtrlType, hook_sigint=thread.interrupt_main):
if dwCtrlType == 0: # CTRL_C_EVENT
hook_sigint()
return 1 # don't chain to the next handler
return 0 # chain to the next handler
win32api.SetConsoleCtrlHandler(handler, 1)
>>> import time
>>> from scipy import stats
>>> time.sleep(10)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
KeyboardInterrupt
[1] interrupt_main calls PyErr_SetInterrupt. This trips Handlers[SIGINT] and calls Py_AddPendingCall to add checksignals_witharg. In turn this calls PyErr_CheckSignals. Since Handlers[SIGINT] is tripped, this calls Handlers[SIGINT].func. Finally, if func is signal.default_int_handler, you'll get a KeyboardInterrupt exception.
Setting the environment variable FOR_DISABLE_CONSOLE_CTRL_HANDLER to 1 seems to fix the issue, but only if it is set before loading offending packages.
import os
os.environ['FOR_DISABLE_CONSOLE_CTRL_HANDLER'] = '1'
[...]
EDIT: While Ctrl+C doesn't crash python anymore, it also fails to stop the current calculation.
I have been able to get a half-workaround by doing this:
from scipy import stats
import win32api
def doSaneThing(sig, func=None):
return True
win32api.SetConsoleCtrlHandler(doSaneThing, 1)
Returning true in the handler stops the chain of handlers so that the meddling Fortran handler is no longer called. However, this workaround is only partial, for two reasons:
It does not actually raise a KeyboardInterrupt, meaning that I can't react to it in Python code. It just drops me back to the prompt.
It doesn't fully interrupt things in the way that Ctrl-C normally does in Python. If in a fresh Python session I do a time.sleep(3) and hit Ctrl-C, the sleep is immediately aborted and I get a KeyboardInterrupt. With the above workaround, the sleep is not aborted, and control returns to the prompt only after the sleep time is up.
Nonetheless, this is still better than crashing the whole session. To me this raises the question of why SciPy (and any other Python libraries that rely on these Intel libraries) don't do this themselves.
I'm leaving this answer unaccepted in the hope that someone can provide a real solution or workaround. By "real" I mean that pressing Ctrl-C during a long-running SciPy calculation should work just like it does when SciPy is not loaded. (Note that this doesn't mean it has to work immediately. Non-SciPy calculations like plain Python sum(xrange(100000000)) may not immediately abort on Ctrl-C, but at least when they do, they raise a KeyboardInterrupt.)
Here's code to patch the dll to remove the call that installs the Ctrl-C handler:
import os
import os.path
import imp
import hashlib
basepath = imp.find_module('numpy')[1]
ifcoremd = os.path.join(basepath, 'core', 'libifcoremd.dll')
with open(ifcoremd, 'rb') as dll:
contents = dll.read()
m = hashlib.md5()
m.update(contents)
patch = {'7cae928b035bbdb90e4bfa725da59188': (0x317FC, '\xeb\x0b'),
'0f86dcd44a1c2e217054c50262f727bf': (0x3fdd9, '\xeb\x10')}[m.hexdigest()]
if patch:
contents = bytearray(contents)
contents[patch[0]:patch[0] + len(patch[1])] = patch[1]
with open(ifcoremd, 'wb') as dll:
dll.write(contents)
else:
print 'Unknown dll version'
EDIT: Here's how I added a patch for the x64. Run python.exe in the debugger, and set a breakpoint for SetConsoleCtrlHandler until you get to the call you want to patch out:
Microsoft (R) Windows Debugger Version 6.12.0002.633 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
CommandLine: .\venv\Scripts\python.exe
...
0:000> .symfix
0:000> bp kernel32!SetConsoleCtrlHandler
0:000> g
Breakpoint 0 hit
KERNEL32!SetConsoleCtrlHandler:
00007ffc`c25742f0 ff252af00400 jmp qword ptr [KERNEL32!_imp_SetConsoleCtrlHandler (00007ffc`c25c3320)] ds:00007ffc`c25c3320={KERNELBASE!SetConsoleCtrlHandler (00007ffc`bfa12e10)}
0:000> k 5
Child-SP RetAddr Call Site
00000000`007ef7a8 00000000`71415bb4 KERNEL32!SetConsoleCtrlHandler
*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\WINDOWS\SYSTEM32\python27.dll -
00000000`007ef7b0 00000000`7035779f MSVCR90!signal+0x17c
00000000`007ef800 00000000`70237ea7 python27!PyOS_getsig+0x3f
00000000`007ef830 00000000`703546cc python27!Py_Main+0x21ce7
00000000`007ef880 00000000`7021698c python27!Py_InitializeEx+0x40c
0:000> g
Python 2.7.11 (v2.7.11:6d1b6a68f775, Dec 5 2015, 20:40:30) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
...
Breakpoint 0 hit
KERNEL32!SetConsoleCtrlHandler:
00007ffc`c25742f0 ff252af00400 jmp qword ptr [KERNEL32!_imp_SetConsoleCtrlHandler (00007ffc`c25c3320)] ds:00007ffc`c25c3320={KERNELBASE!SetConsoleCtrlHandler (00007ffc`bfa12e10)}
0:000> k 5
Child-SP RetAddr Call Site
00000000`007ec308 00000000`7023df6e KERNEL32!SetConsoleCtrlHandler
00000000`007ec310 00000000`70337877 python27!PyTime_DoubleToTimet+0x10ee
00000000`007ec350 00000000`7033766d python27!PyImport_IsScript+0x4f7
00000000`007ec380 00000000`70338bf2 python27!PyImport_IsScript+0x2ed
00000000`007ec3b0 00000000`703385a9 python27!PyImport_ImportModuleLevel+0xc82
0:000> g
...
>>> import scipy.stats
...
Breakpoint 0 hit
KERNEL32!SetConsoleCtrlHandler:
00007ffc`c25742f0 ff252af00400 jmp qword ptr [KERNEL32!_imp_SetConsoleCtrlHandler (00007ffc`c25c3320)] ds:00007ffc`c25c3320={KERNELBASE!SetConsoleCtrlHandler (00007ffc`bfa12e10)}
0:000> k 5
*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\Users\kevin\Documents\\venv\lib\site-packages\numpy\core\libifcoremd.dll -
Child-SP RetAddr Call Site
00000000`007ed818 00007ffc`828309eb KERNEL32!SetConsoleCtrlHandler
00000000`007ed820 00007ffc`828dfa44 libifcoremd!GETEXCEPTIONPTRSQQ+0xdb
00000000`007ed880 00007ffc`828e59d7 libifcoremd!for_lt_ne+0xc274
00000000`007ed8b0 00007ffc`828e5aff libifcoremd!for_lt_ne+0x12207
00000000`007ed8e0 00007ffc`c292ddc7 libifcoremd!for_lt_ne+0x1232f
0:000> ub 00007ffc`828309eb
libifcoremd!GETEXCEPTIONPTRSQQ+0xbb:
00007ffc`828309cb 00e8 add al,ch
00007ffc`828309cd df040b fild word ptr [rbx+rcx]
00007ffc`828309d0 0033 add byte ptr [rbx],dh
00007ffc`828309d2 c9 leave
00007ffc`828309d3 ff15bf390e00 call qword ptr [libifcoremd!for_lt_ne+0x40bc8 (00007ffc`82914398)]
00007ffc`828309d9 488d0d00efffff lea rcx,[libifcoremd!for_rtl_finish_+0x20 (00007ffc`8282f8e0)]
00007ffc`828309e0 ba01000000 mov edx,1
00007ffc`828309e5 ff158d390e00 call qword ptr [libifcoremd!for_lt_ne+0x40ba8 (00007ffc`82914378)]
We'll patch out the lea instruction with a relative jmp (which is 0xeb followed by the number of bytes to jump)
0:000> ? 00007ffc`828309eb - 00007ffc`828309d9
Evaluate expression: 18 = 00000000`00000012
0:000> f 00007ffc`828309d9 L2 eb 10
Filled 0x2 bytes
0:000> ub 00007ffc`828309eb
libifcoremd!GETEXCEPTIONPTRSQQ+0xbe:
00007ffc`828309ce 040b add al,0Bh
00007ffc`828309d0 0033 add byte ptr [rbx],dh
00007ffc`828309d2 c9 leave
00007ffc`828309d3 ff15bf390e00 call qword ptr [libifcoremd!for_lt_ne+0x40bc8 (00007ffc`82914398)]
00007ffc`828309d9 eb10 jmp libifcoremd!GETEXCEPTIONPTRSQQ+0xdb (00007ffc`828309eb)
00007ffc`828309db 0d00efffff or eax,0FFFFEF00h
00007ffc`828309e0 ba01000000 mov edx,1
00007ffc`828309e5 ff158d390e00 call qword ptr [libifcoremd!for_lt_ne+0x40ba8 (00007ffc`82914378)]
I don't know how the .dll file is mapped in this process, so I'll just search for 0d 00 ef ff ff in the file with a hex editor. It is a unique hit, so we can calculate the location in the .dll to patch.
0:000> db 00007ffc`828309d0
00007ffc`828309d0 00 33 c9 ff 15 bf 39 0e-00 eb 10 0d 00 ef ff ff .3....9.........
00007ffc`828309e0 ba 01 00 00 00 ff 15 8d-39 0e 00 48 8d 0d 0e 9c ........9..H....
00007ffc`828309f0 09 00 e8 09 2e 0a 00 48-8d 0d 32 9f 09 00 e8 fd .......H..2.....
00007ffc`82830a00 2d 0a 00 48 8d 0d ca ee-0e 00 e8 51 90 00 00 85 -..H.......Q....
00007ffc`82830a10 c0 0f 85 88 02 00 00 e8-38 fa 0a 00 ff 15 4e 39 ........8.....N9
00007ffc`82830a20 0e 00 89 c1 e8 d7 2d 0a-00 48 8d 05 f8 be 11 00 ......-..H......
00007ffc`82830a30 45 32 e4 c7 05 0b 4a 13-00 00 00 00 00 41 bd 01 E2....J......A..
00007ffc`82830a40 00 00 00 48 89 05 06 4a-13 00 ff 15 30 39 0e 00 ...H...J....09..
0:000> ? 00007ffc`828309d9 - 00007ffc`828309d0
Evaluate expression: 9 = 00000000`00000009
0:000> ? 00007ffc`828309d9 - 00007ffc`828309d0 + 3FDD0
Evaluate expression: 261593 = 00000000`0003fdd9
0:000>
Ok, I've patched the dll at 0x3fdd9. Let's see what it looks like now:
Microsoft (R) Windows Debugger Version 6.12.0002.633 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
CommandLine: .\venv\Scripts\python.exe
...
0:000> bp libifcoremd!GETEXCEPTIONPTRSQQ+c9
Bp expression 'libifcoremd!GETEXCEPTIONPTRSQQ+c9' could not be resolved, adding deferred bp
0:000> g
Python 2.7.11 (v2.7.11:6d1b6a68f775, Dec 5 2015, 20:40:30) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import scipy.stats
...
Breakpoint 0 hit
libifcoremd!GETEXCEPTIONPTRSQQ+0xc9:
00007ffc`845909d9 eb10 jmp libifcoremd!GETEXCEPTIONPTRSQQ+0xdb (00007ffc`845909eb)
0:000> u
libifcoremd!GETEXCEPTIONPTRSQQ+0xc9:
00007ffc`845909d9 eb10 jmp libifcoremd!GETEXCEPTIONPTRSQQ+0xdb (00007ffc`845909eb)
00007ffc`845909db 0d00efffff or eax,0FFFFEF00h
00007ffc`845909e0 ba01000000 mov edx,1
00007ffc`845909e5 ff158d390e00 call qword ptr [libifcoremd!for_lt_ne+0x40ba8 (00007ffc`84674378)]
00007ffc`845909eb 488d0d0e9c0900 lea rcx,[libifcoremd!GETHANDLEQQ (00007ffc`8462a600)]
00007ffc`845909f2 e8092e0a00 call libifcoremd!for_lt_ne+0x30 (00007ffc`84633800)
00007ffc`845909f7 488d0d329f0900 lea rcx,[libifcoremd!GETUNITQQ (00007ffc`8462a930)]
00007ffc`845909fe e8fd2d0a00 call libifcoremd!for_lt_ne+0x30 (00007ffc`84633800)
0:000>
So now were are jmping over pushing the arguments on the stack and the function call. So its Ctrl-C handler will not be installed.
Workaround: patch SetControlCtrlHandler
import ctypes
SetConsoleCtrlHandler_body_new = b'\xC2\x08\x00' if ctypes.sizeof(ctypes.c_void_p) == 4 else b'\xC3'
try: SetConsoleCtrlHandler_body = (lambda kernel32: (lambda pSetConsoleCtrlHandler:
kernel32.VirtualProtect(pSetConsoleCtrlHandler, ctypes.c_size_t(1), 0x40, ctypes.byref(ctypes.c_uint32(0)))
and (ctypes.c_char * 3).from_address(pSetConsoleCtrlHandler.value)
)(ctypes.cast(kernel32.SetConsoleCtrlHandler, ctypes.c_void_p)))(ctypes.windll.kernel32)
except: SetConsoleCtrlHandler_body = None
if SetConsoleCtrlHandler_body:
SetConsoleCtrlHandler_body_old = SetConsoleCtrlHandler_body[0:len(SetConsoleCtrlHandler_body_new)]
SetConsoleCtrlHandler_body[0:len(SetConsoleCtrlHandler_body_new)] = SetConsoleCtrlHandler_body_new
try:
import scipy.stats
finally:
if SetConsoleCtrlHandler_body:
SetConsoleCtrlHandler_body[0:len(SetConsoleCtrlHandler_body_new)] = SetConsoleCtrlHandler_body_old
This worked for me:
import os
os.environ['FOR_DISABLE_CONSOLE_CTRL_HANDLER'] = '1'
from scipy.stats import zscore
Try
import os
os.environ['FOR_IGNORE_EXCEPTIONS'] = '1'
import scipy.stats
Related
I have a python function that uses ctypes to call some c code
The python code looks like:
import numpy as np
from ctypes import *
dll=cdll.LoadLibrary("./test.so")
def callTestFunction():
out=np.zeros(shape=(10),dtype=np.float32)
dll.testFunction(out.ctypes.data,10)
return out
And the c code looks like
void testFunction(float values[], int l){
for(int i=0;i<l;i++){
values[i]=1;
}
}
This code works fine if I run python and call the function.
But when I import the same code and call the function inside of mod_wsgi, I get a segmentation fault.
I have already set WSGIApplicationGroup %{GLOBAL} and if I capture the segfault in httpd using gdb, it says
#0 0x00007fffc9fa9bad in testFunction (values=0x563e93b0, l=10) at test.c:63
63 values[i]=1;
(gdb) print *values#10
Cannot access memory at address 0x563e93b0
My guess is Apache is enforcing some kind of memory boundary between my python code and my c library? Does anyone have a solution for this, or know of a better way to return an array from c to python inside of mod_wsgi?
Update:
I added print statements to python and c to print out
sizeof(c_void_p), out.ctypes.data, and values.
in ipython
out.ctypes.data 0x23dde40
sizeof(c_void_p) 8
values (in c): 23dde40
in apache
[Sun May 10 17:37:01.647440 2020] [wsgi:error] [pid 7101] [client 127.0.0.1:60346] out.ctypes.data 0x55555645f7c0
[Sun May 10 17:37:01.647592 2020] [wsgi:error] [pid 7101] [client 127.0.0.1:60346] sizeof(c_void_p) 8
...
(gdb) p values
$1 = (float *) 0x5645f7c0
So there is a difference in Apache! 0x55555645f7c0 vs 0x5645f7c0
If I look at the correct memory location in GDB, it looks promising!
(gdb) p *0x55555645f7c0#40
$2 = {0 <repeats 40 times>}
Turns out I need to cast out.ctypes.data as c_void_p!
Corrected python code:
import numpy as np
from ctypes import *
dll=cdll.LoadLibrary("./test.so")
def callTestFunction():
out=np.zeros(shape=(10),dtype=np.float32)
dll.testFunction(c_void_p(out.ctypes.data),c_int(10))
return out
I still have no idea why this works in ipython, but not Apache.
I'm working on a dump, which I try to investigate, using PYKD technology.
The result of the x /2 *!*``vtable' (just one backtick) contains the following result:
745293b8 mfc110u!CPtrList::`vftable'
However, when I try to get more information about this class, I get a "symbol not found" exception:
Python source code:
dprintln("name=[%s]" % type_stats.name)
if not type_stats.name in typesize_by_type:
try:
type_info = typeInfo(type_stats.name)
except Exception, e:
dprintln("text=[%s]" % (str(e)))
Output:
name=[mfc110u!CPtrList]
text=['CPtrList' - symbol not found]
The result of the lm command returns the mfc110u symbols, as you can see here:
0:000> lm
start end module name
...
744f0000 74930000 mfc110u (pdb symbols) C:\ProgramData\dbg\sym\mfc110u.i386.pdb\...\mfc110u.i386.pdb
...
For your information, I'm now working with the last version of PYKD:
0:000> .chain
Extension DLL search Path:
...
Extension DLL chain:
pykd.pyd: image 0.3.3.4, API 1.0.0, built Mon May 14 11:14:43 2018
[path: C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\winext\pykd.pyd]
Meanwhile I've found a very simple way for reproducing the issue without needing to launch the whole script (using the Windbg prompt):
0:000> !py
Python 2.7.10 (default, May 23 2015, 09:40:32) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> typeInfo("mfc110u!CPtrList")
Traceback (most recent call last):
File "<console>", line 1, in <module>
SymbolException: 'CPtrList' - symbol not found
In addition to ussrhero's answer, there is following extra information:
The result of x /2 *!CPtrList* contains (amongst many others) the following results:
009530c4 <application>!CPtrList::~CPtrList
009530be <application>!CPtrList::CPtrList
... <application>!CPtrList::...
009abc5c <application>!CPtrList::`RTTI Base Class Array'
009abc4c <application>!CPtrList::`RTTI Class Hierarchy Descriptor'
009bcd18 <application>!CPtrList `RTTI Type Descriptor'
009abc30 <application>!CPtrList::`RTTI Base Class Descriptor at (0,-1,0,64)'
7464e9cb mfc110u!CPtrList::~CPtrList
74544a04 mfc110u!CPtrList::CPtrList
... mfc110u!CPtrList::...
745293b8 mfc110u!CPtrList::`vftable'
747510da mfc110u!CPtrList::`vector deleting destructor'
745293cc mfc110u!CPtrList::`RTTI Complete Object Locator'
7452940c mfc110u!CPtrList::`RTTI Base Class Array'
745293fc mfc110u!CPtrList::`RTTI Class Hierarchy Descriptor'
74795778 mfc110u!CPtrList `RTTI Type Descriptor'
745293e0 mfc110u!CPtrList::`RTTI Base Class Descriptor at (0,-1,0,64)'
746fdc68 mfc110u!CPtrList::classCPtrList
The script I'm using (heap_stat.py) browses through the results of !heap -h 0 and searches for the entry, corresponding with mfc110u!CPtrList::``vtable'.
The result of dt CPtrList starts with the following:
0:000> dt CPtrList
<application>!CPtrList => in other words, no 'mfcu110' entry
+0x000 __VFN_table : Ptr32
I'm already wondering for a long time, what's the difference between the mfc110u!CPtrList and the <application>!CPtrList entries and what's the exact role of the vtable entry in x /2 result?
Any ideas?
Thanks
Meanwhile I've found the solution:
Apparently for some objects, the module prefix needs to be removed:
>>> typeInfo("mdf110u!CPtrList")
-> SymbolException
>>> typeInfo("CPtrList")
-> this is working!!!
Try to see how WinDBG locates this type:
dt CPtrList
It maybe mfc110u does not contain type information fot CPtrList
I'm trying to use python 2.7.5 in a c/c++ DLL. This DLL is used by another application which has made debugging a challenge. After banging away at this for hours I have isolated the problem down to where a file read in a 'with' statement is throwing an exception. This I do not understand...'with' should absorb an exception if implemented correctly, right? Anyway calling the same python code from the command line has no problems at all.
My C/CPP DLL calls this...
def parsetest(filename):
bytesin = getMD3Bytes(filename)
return bytesin
def getMD3Bytes(filename):
filename = 'lower.md3'
bytes = ''
valuetoreturn = 1
try:
with open(filename,'rb') as fileptr:
if fileptr != None:
bytes = fileptr.read()
valuetoreturn = 333
except:
valuetoreturn = 991
return valuetoreturn
If the DLL runs this code via...
pValue = PyObject_CallObject(pFunc, arguments);
And gets a result via...
iResult = PyInt_AsLong(pValue);
iResult has the value of 991 instead of 333 which would only happen if an exception had not occurred inside of 'with'. I know because I had the app calling the DLL pop up a message box with iResult in it.
Even more interesting to me, this works...
C:\Program Files (x86)\DeleD CE\Plugins>python
Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import parseMD3
>>> testval = parseMD3.parsetest('junk')
>>> print testval
333
>>> exit()
So why does the CLI python return a different result that the same code being call from PyObject_CallObject? Why does 'with' behave differently here?
with does not handle exceptions. It only ensures that the file is closed when an exception occurs. If an exception occurs in the open() expression itself, the with block isn't even entered; fileptr will never be bound to None either.
You are catching all exceptions, including keyboard interrupts and memory errors, so we cannot begin to tell why the code fails when running under C++ control here.
Stick to a limited set of exceptions instead, like IOError, and log the exception properly:
import logging
logger = logging.getLogger('__name__')
def getMD3Bytes(filename):
filename = 'lower.md3'
bytes = ''
valuetoreturn = 1
try:
with open(filename,'rb') as fileptr:
bytes = fileptr.read()
valuetoreturn = 333
except IOError:
logger.exception('Failed to open file properly')
valuetoreturn = 991
return valuetoreturn
The default config for the logger will output to stderr, but you can configure it to log to a file instead:
logging.basicConfig(filename='/tmp/debug.log', level=logging.DEBUG)
I normally have no problem scouring the web for obscure python packages. However I cannot find this one. Does anyone know what the error is referring too?
Is there an easy way to dig into the source to figure out what modules it wants?
I know I shouldn't be bugging stackoverflow about every missing module I have so is there a method for debugging this myself?
These are the modules i've most recently installed for this project:
blist
beautifulsoup
requests
pymongo
SQLAlchemy
xmltodict
financial_fundamentals
I think the error is coming from SQLAlchemy but the error readout says something about vector_cache
Using the following line:
import financial_fundamentals.accounting_metrics as ac
I get:
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-6-f1aecf47aa1b> in <module>()
----> 1 import financial_fundamentals.accounting_metrics as ac
C:\Python27\lib\site-packages\financialfundamentals-0.2.3-py2.7.egg\financial_fundamentals\accounting_metrics.py in <module>()
65 import financial_fundamentals.edgar as edgar
66
---> 67 #vector_cache.vector_cache
68 def earnings_per_share(required_data):
69 start, end = required_data.index[0], required_data.index[-1]
C:\Python27\lib\site-packages\vector_cache-0.1.0-py2.7.egg\vector_cache\_vector_cache.pyc in vector_cache(user_function)
19 # When the function name or location changes you're going to have to re-cache everything, bummer.
20 metric = user_function.__name__
---> 21 data_store = get_data_store()
22 #wraps(user_function)
23 def wrapper(required_data_df):
C:\Python27\lib\site-packages\vector_cache-0.1.0-py2.7.egg\vector_cache\__init__.py in _get_data_store(cls)
37 except IOError:
38 # no config in home directory
---> 39 return cls._default_data_store(db_dir=home)
40 else:
41 return cls._eval_config_code(code)
C:\Python27\lib\site-packages\vector_cache-0.1.0-py2.7.egg\vector_cache\__init__.py in _default_data_store(db_dir)
43 #staticmethod
44 def _default_data_store(db_dir):
---> 45 from sql_driver import SQLDataStore
46 import os
47 db_file_path = os.path.join(db_dir, 'vector_cache.db')
ImportError: No module named sql_driver
...for current user? for all users?
I'm working an a small program which needs to create links in the start menu. Currently I'm hardcoding like below, but it only works in english locales, for example it should be "Startmenü" in german. What are cleaner, more portable approaches?
OUR_STARTMENU = os.environ['ALLUSERSPROFILE'] + '\Start Menu\Programs\Our Stuff'
thank you
I've heard of 2 ways of doing this. First:
from win32com.shell import shell
shell.SHGetSpecialFolderPath(0,shellcon.CSIDL_COMMON_STARTMENU)
Second, using the WScript.Shell object (source : http://www.mail-archive.com/python-win32#python.org/msg00992.html):
import win32com.client
objShell = win32com.client.Dispatch("WScript.Shell")
allUserProgramsMenu = objShell.SpecialFolders("AllUsersPrograms")
userMenu = objShell.SpecialFolders("StartMenu")
Another source: http://blogs.msdn.com/saveenr/archive/2005/12/28/creating-a-start-menu-shortcut-with-powershell-and-python.aspx
Also, CSIDL_COMMON_STARTMENU is for all user startup and CSIDL_STARTMENU for current user startup.
A friend, Luke Pinner of Environment.gov.au, gave a solution by email which uses a core module (python 2.5+). Believed to be multi-lingual as the return from the API call is unicode. Tested on Win7 with Japanese locale, and on another us-english machine by manually changing Start Menu to point to %USERPROFILE%\Startmenü
''' Get windows special folders without pythonwin
Example:
import specialfolders
start_programs = specialfolders.get(specialfolders.PROGRAMS)
Code is public domain, do with it what you will.
Luke Pinner - Environment.gov.au, 2010 February 10
'''
#Imports use _syntax to mask them from autocomplete IDE's
import ctypes as _ctypes
from ctypes.wintypes import HWND as _HWND, HANDLE as _HANDLE,DWORD as _DWORD,LPCWSTR as _LPCWSTR,MAX_PATH as _MAX_PATH, create_unicode_buffer as _cub
_SHGetFolderPath = _ctypes.windll.shell32.SHGetFolderPathW
#public special folder constants
DESKTOP= 0
PROGRAMS= 2
MYDOCUMENTS= 5
FAVORITES= 6
STARTUP= 7
RECENT= 8
SENDTO= 9
STARTMENU= 11
MYMUSIC= 13
MYVIDEOS= 14
NETHOOD= 19
FONTS= 20
TEMPLATES= 21
ALLUSERSSTARTMENU= 22
ALLUSERSPROGRAMS= 23
ALLUSERSSTARTUP= 24
ALLUSERSDESKTOP= 25
APPLICATIONDATA= 26
PRINTHOOD= 27
LOCALSETTINGSAPPLICATIONDATA= 28
ALLUSERSFAVORITES= 31
LOCALSETTINGSTEMPORARYINTERNETFILES=32
COOKIES= 33
LOCALSETTINGSHISTORY= 34
ALLUSERSAPPLICATIONDATA= 35
def get(intFolder):
_SHGetFolderPath.argtypes = [_HWND, _ctypes.c_int, _HANDLE, _DWORD, _LPCWSTR]
auPathBuffer = _cub(_MAX_PATH)
exit_code=_SHGetFolderPath(0, intFolder, 0, 0, auPathBuffer)
return auPathBuffer.value