EDIT (begin)
After spending time with #eryksun we dug really deep and found the underlying issue. These two examples fail1.py and fail2.py have the same effect as the original lengthier example I posted.
It turns out the issue has to do with copying 4-byte C ints into an 8-byte stack location without zero'ing out the memory first, potentially leaving garbage in the upper 4 bytes. This was confirmed using a debugger (again props to #eryksun).
This was one of those weird bugs (heisenbug) where you can add some printf statements then the bug doesn't exist any more.
the fix
At the top ffi_prep_args on the line before argp = stack;, add a call
to memset(stack, 0, ecif->cif->bytes);. This will zero the entire
stack.
This is the location to fix:
https://github.com/python/cpython/blob/v2.7.13/Modules/_ctypes/libffi_msvc/ffi.c#L47
fail1.py
import ctypes
kernel32 = ctypes.WinDLL('kernel32')
hStdout = kernel32.GetStdHandle(-11)
written = ctypes.c_ulong()
kernel32.WriteFile(hStdout, b'spam\n', 5, ctypes.byref(written), None)
fail2.py
import ctypes
import msvcrt
import sys
kernel32 = ctypes.WinDLL('kernel32')
hStdout = msvcrt.get_osfhandle(sys.stdout.fileno())
written = ctypes.c_ulong()
kernel32.WriteFile(hStdout, b'spam\n', 5, ctypes.byref(written), None)
EDIT (end)
I built my own Python 2.7.13 for Windows because I'm using bindings I created to a 3rd party application which must be built with a specific build of Visual Studio 2012.
I started to code up some stuff using the "click" module and found an error when using click.echo with any kind of unicode echo.
Python 2.7.13 (default, Mar 27 2017, 11:11:01) [MSC v.1700 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import click
>>> click.echo('Hello World')
Hello World
>>> click.echo(u'Hello World')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\eric\my_env\lib\site-packages\click\utils.py", line 259, in echo
file.write(message)
File "C:\Users\eric\my_env\lib\site-packages\click\_winconsole.py", line 180, in write
return self._text_stream.write(x)
File "C:\Users\eric\my_env\lib\site-packages\click\_compat.py", line 63, in write
return io.TextIOWrapper.write(self, x)
File "C:\Users\eric\my_env\lib\site-packages\click\_winconsole.py", line 164, in write
raise OSError(self._get_error_message(GetLastError()))
OSError: Windows error 6
If I download and install the Python 2.7.13 64-bit installer I don't get this issue. It echo's just fine.
I have looked into this a lot and am at a loss right now.
I'm not too familiar with Windows, Visual Studio, or ctypes.
I spent some time looking at the code path to produce the smallest file (without click) which demonstrates this problem (see below)
It produces the same "Windows error 6"... again, this works fine with the python installed from the 2.7.13 64 bit MSI installer.
Can someone share the process used to create the Windows installers? Is this a manual process or is it automated?
Maybe I'm missing some important switch to msbuild or something. Any help or ideas are appreciated.
I cannot use a downloaded copy of Python... it needs to be built with a specific version, update, patch, etc of Visual Studio.
All I did was
clone cpython from github and checkout 2.7.13
edit some xp stuff out of tk stuff to get it to compile on Windows Server 2003
In externals\tk-8.5.15.0\win\Makefile.in remove ttkWinXPTheme.$(OBJEXT) line
In externals\tk-8.5.15.0\win\makefile.vc remove $(TMP_DIR)\ttkWinXPTheme.obj line
In externals\tk-8.5.15.0\win\ttkWinMonitor.c remove 2 TtkXPTheme_Init lines
In PCbuild\tcltk.props change VC9 to VC11 at the bottom
PCbuild\build.bat -e -p x64 "/p:PlatformToolset=v110"
After that I created an "install" by copying .exe, .pyd, .dll files, ran get-pip.py, then python -m pip install virtualenv, then virtualenv my_env, then activated it, then did a pip install click.
But with this stripped down version you don't need pip, virtualenv or click... just ctypes.
You could probably even build it without the -e switch to build.bat.
from ctypes import byref, POINTER, py_object, pythonapi, Structure, windll
from ctypes import c_char, c_char_p, c_int, c_ssize_t, c_ulong, c_void_p
c_ssize_p = POINTER(c_ssize_t)
kernel32 = windll.kernel32
STDOUT_HANDLE = kernel32.GetStdHandle(-11)
PyBUF_SIMPLE = 0
MAX_BYTES_WRITTEN = 32767
class Py_buffer(Structure):
_fields_ = [
('buf', c_void_p),
('obj', py_object),
('len', c_ssize_t),
('itemsize', c_ssize_t),
('readonly', c_int),
('ndim', c_int),
('format', c_char_p),
('shape', c_ssize_p),
('strides', c_ssize_p),
('suboffsets', c_ssize_p),
('internal', c_void_p)
]
_fields_.insert(-1, ('smalltable', c_ssize_t * 2))
bites = u"Hello World".encode('utf-16-le')
bytes_to_be_written = len(bites)
buf = Py_buffer()
pythonapi.PyObject_GetBuffer(py_object(bites), byref(buf), PyBUF_SIMPLE)
buffer_type = c_char * buf.len
buf = buffer_type.from_address(buf.buf)
code_units_to_be_written = min(bytes_to_be_written, MAX_BYTES_WRITTEN) // 2
code_units_written = c_ulong()
kernel32.WriteConsoleW(STDOUT_HANDLE, buf, code_units_to_be_written, byref(code_units_written), None)
bytes_written = 2 * code_units_written.value
if bytes_written == 0 and bytes_to_be_written > 0:
raise OSError('Windows error %s' % kernel32.GetLastError())
Related
my code..
import ctypes
import win32security
h_token = win32security.OpenProcessToken(ctypes.windll.kernel32.GetCurrentProcess(), win32security.TOKEN_ALL_ACCESS)
lpApplicationName = ctypes.c_wchar_p(rf"C:\\Windows\\System32\\cmd.exe")
lpCommandLine = ctypes.c_wchar_p("")
dwCreationFlags = 0x00000010
lpEnvironment = None
lpProcessAttributes = None
lpThreadAttributes = None
bInheritHandles = False
ctypes.windll.advapi32.CreateProcessWithTokenW(h_token, 0, lpApplicationName, lpCommandLine, dwCreationFlags, lpEnvironment, None, lpProcessAttributes, lpThreadAttributes, bInheritHandles)
my output...
Traceback (most recent call last):
File "testx.py", line 96, in <module>
ctypes.windll.advapi32.CreateProcessWithTokenW(h_token, 0, lpApplicationName, lpCommandLine, dwCreationFlags, lpEnvironment, None, lpProcessAttributes, lpThreadAttributes, bInheritHandles)
ctypes.ArgumentError: argument 1: TypeError: Don't know how to convert parameter 1
how i can fix it? what i'm doing wrong?, thanks for read and help :)
It's not a good idea to mix libraries / tools / frameworks, when things can be done in one of them. In this case:
[GitHub]: mhammond/pywin32 - Python for Windows (pywin32) Extensions is a Python wrapper over WinAPIs. Documentation (WiP) can be found at [GitHub.MHammond]: Python for Win32 Extensions Help (or [ME.TimGolden]: Python for Win32 Extensions Help)
[Python.Docs]: ctypes - A foreign function library for Python
At 1st glance, this situation seems to be one of the exceptions, as CreateProcessWithTokenW is not wrapped by PyWin32.
I played a bit with your code after fixing some errors:
Pass h_token.handle to CreateProcessWithTokenW
Fix Undefined Behavior (check [SO]: C function called from Python via ctypes returns incorrect value (#CristiFati's answer) for a common pitfall when working with CTypes (calling functions))
but I couldn't get it to work (got ERROR_TOKEN_ALREADY_IN_USE). Note that I didn't spend much time investigating (adjusting the token privileges, ...), because I paid more attention to what [MS.Learn]: CreateProcessWithTokenW function (winbase.h) states:
The process that calls CreateProcessWithTokenW must have the SE_IMPERSONATE_NAME privilege. If this function fails with ERROR_PRIVILEGE_NOT_HELD (1314), use the CreateProcessAsUser or CreateProcessWithLogonW function instead.
code00.py:
#!/usr/bin/env python
import sys
import win32api as wapi
import win32con as wcon
import win32process as wproc
import win32security as wsec
def main(*argv):
token = wsec.OpenProcessToken(wproc.GetCurrentProcess(), wsec.TOKEN_ALL_ACCESS)
#print(wsec.GetTokenInformation(token, wsec.TokenType))
app_name = "C:\\Windows\\System32\\cmd.exe"
creation_flags = wcon.CREATE_NEW_CONSOLE
si = wproc.STARTUPINFO()
hproc, hthr, pid, tid = wproc.CreateProcessAsUser(token, app_name, None,
None, None, 0, creation_flags,
None, None, si)
print("New PId: {:d}".format(pid))
wapi.CloseHandle(token)
if __name__ == "__main__":
print("Python {:s} {:03d}bit on {:s}\n".format(" ".join(elem.strip() for elem in sys.version.split("\n")),
64 if sys.maxsize > 0x100000000 else 32, sys.platform))
rc = main(*sys.argv[1:])
print("\nDone.\n")
sys.exit(rc)
Output:
[cfati#CFATI-5510-0:e:\Work\Dev\StackOverflow\q075358233]> "e:\Work\Dev\VEnvs\py_pc064_03.10_test0\Scripts\python.exe" ./code00.py
Python 3.10.9 (tags/v3.10.9:1dd9be6, Dec 6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)] 064bit on win32
New PId: 20512
Done.
And a new Cmd window popped up. Note that my user has full administrative (God like) privileges.
Similar situations:
[SO]: Get the title of a window of another program using the process name (#CristiFati's answer)
[SO]: Python3 get process base-address from PID (#CristiFati's answer)
The token returned by win32security.OpenProcessToken is a PyHANDLE object. Use int(h_token) to get a value that ctypes will accept. Note that you should set the .argtypes and .restype of a ctypes function or parameters and return value may not convert properly from Python to C.
Is there a way to tell Pandas to use a specific pickle protocol (e.g. 4) when writing an HDF5 file?
Here is the situation (much simplified):
Client A is using python=3.8.1 (as well as pandas=1.0.0 and pytables=3.6.1). A writes some DataFrame using df.to_hdf(file, key).
Client B is using python=3.7.1 (and, as it happened, pandas=0.25.1 and pytables=3.5.2 --but that's irrelevant). B tries to read the data written by A using pd.read_hdf(file, key), and fails with ValueError: unsupported pickle protocol: 5.
Mind you, this doesn't happen with a purely numerical DataFrame (e.g. pd.DataFrame(np.random.normal(size=(10,10))). So here is a reproducible example:
(base) $ conda activate py38
(py38) $ python
Python 3.8.1 (default, Jan 8 2020, 22:29:32)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> df = pd.DataFrame(['hello', 'world']))
>>> df.to_hdf('foo', 'x')
>>> exit()
(py38) $ conda deactivate
(base) $ python
Python 3.7.4 (default, Aug 13 2019, 20:35:49)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> df = pd.read_hdf('foo', 'x')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/anaconda3/lib/python3.7/site-packages/pandas/io/pytables.py", line 407, in read_hdf
return store.select(key, auto_close=auto_close, **kwargs)
File "/opt/anaconda3/lib/python3.7/site-packages/pandas/io/pytables.py", line 782, in select
return it.get_result()
File "/opt/anaconda3/lib/python3.7/site-packages/pandas/io/pytables.py", line 1639, in get_result
results = self.func(self.start, self.stop, where)
File "/opt/anaconda3/lib/python3.7/site-packages/pandas/io/pytables.py", line 766, in func
return s.read(start=_start, stop=_stop, where=_where, columns=columns)
File "/opt/anaconda3/lib/python3.7/site-packages/pandas/io/pytables.py", line 3206, in read
"block{idx}_values".format(idx=i), start=_start, stop=_stop
File "/opt/anaconda3/lib/python3.7/site-packages/pandas/io/pytables.py", line 2737, in read_array
ret = node[0][start:stop]
File "/opt/anaconda3/lib/python3.7/site-packages/tables/vlarray.py", line 681, in __getitem__
return self.read(start, stop, step)[0]
File "/opt/anaconda3/lib/python3.7/site-packages/tables/vlarray.py", line 825, in read
outlistarr = [atom.fromarray(arr) for arr in listarr]
File "/opt/anaconda3/lib/python3.7/site-packages/tables/vlarray.py", line 825, in <listcomp>
outlistarr = [atom.fromarray(arr) for arr in listarr]
File "/opt/anaconda3/lib/python3.7/site-packages/tables/atom.py", line 1227, in fromarray
return six.moves.cPickle.loads(array.tostring())
ValueError: unsupported pickle protocol: 5
>>>
Note: I tried also reading using pandas=1.0.0 (and pytables=3.6.1) in python=3.7.4. That fails too, so I believe it is simply the Python version (3.8 writer vs 3.7 reader) that causes the problem. This makes sense since pickle protocol 5 was introduced as PEP-574 for Python 3.8.
PyTable uses the highest protocol by default, which is hardcoded here: https://github.com/PyTables/PyTables/blob/50dc721ab50b56e494a5657e9c8da71776e9f358/tables/atom.py#L1216
As a workaround, you can monkey-patch the pickle module on the client A who writes a HDF file. You should do that before importing pandas:
import pickle
pickle.HIGHEST_PROTOCOL = 4
import pandas
df.to_hdf(file, key)
Now the HDF file has been created using pickle protocol version 4 instead version 5.
Update: I was wrong to assume this was not possible. In fact, based on the excellent "monkey-patch" suggestion of #PiotrJurkiewicz, here is a simple context manager that lets us temporarily change the highest pickle protocol. It:
Hides the monkey-patching, and
Has no side-effect outside of the context; it can be used at any time, whether pickle was previously imported or not, before or after pandas, no matter.
Here is the code (e.g. in a file pickle_prot.py):
import importlib
import pickle
class PickleProtocol:
def __init__(self, level):
self.previous = pickle.HIGHEST_PROTOCOL
self.level = level
def __enter__(self):
importlib.reload(pickle)
pickle.HIGHEST_PROTOCOL = self.level
def __exit__(self, *exc):
importlib.reload(pickle)
pickle.HIGHEST_PROTOCOL = self.previous
def pickle_protocol(level):
return PickleProtocol(level)
Usage example in a writer:
import pandas as pd
from pickle_prot import pickle_protocol
pd.DataFrame(['hello', 'world']).to_hdf('foo_0.h5', 'x')
with pickle_protocol(4):
pd.DataFrame(['hello', 'world']).to_hdf('foo_1.h5', 'x')
pd.DataFrame(['hello', 'world']).to_hdf('foo_2.h5', 'x')
And, using a simple test reader:
import pandas as pd
from glob import glob
for filename in sorted(glob('foo_*.h5')):
try:
df = pd.read_hdf(filename, 'x')
print(f'could read {filename}')
except Exception as e:
print(f'failed on {filename}: {e}')
Now, trying to read in py37 after having written in py38, we get:
failed on foo_0.h5: unsupported pickle protocol: 5
could read foo_1.h5
failed on foo_2.h5: unsupported pickle protocol: 5
But, using the same version (37 or 38) to read and write, we of course get no exception.
Note: the issue 33087 is still on Pandas issue tracker.
I'm (was) facing the same problem... I "know" how to solve it and I think you do too...
The solution is to reprocess the whole data to a pickle (or csv) and re-transform it in python3.7 to a hdf5 (which only knows protocol 4).
the flow is something like this:
python3.8 -> hdf5 -> python3.8 -> csv/pickle -> python3.7 -> hdf5 (compatible with both versions)
I avoided this route because I have chuncks of data of a dataframe being exported, creating a large number of files.
Are you actually limited to use python3.7 ? I was limited by tensorflow which as of now only supports up to 3.7 (officially) but you can install tensorflow- nightly-build and it works with python 3.8
Check if you can make the move to 3.8 that would surely solve your problem. :)
I'm working on a dump, which I try to investigate, using PYKD technology.
The result of the x /2 *!*``vtable' (just one backtick) contains the following result:
745293b8 mfc110u!CPtrList::`vftable'
However, when I try to get more information about this class, I get a "symbol not found" exception:
Python source code:
dprintln("name=[%s]" % type_stats.name)
if not type_stats.name in typesize_by_type:
try:
type_info = typeInfo(type_stats.name)
except Exception, e:
dprintln("text=[%s]" % (str(e)))
Output:
name=[mfc110u!CPtrList]
text=['CPtrList' - symbol not found]
The result of the lm command returns the mfc110u symbols, as you can see here:
0:000> lm
start end module name
...
744f0000 74930000 mfc110u (pdb symbols) C:\ProgramData\dbg\sym\mfc110u.i386.pdb\...\mfc110u.i386.pdb
...
For your information, I'm now working with the last version of PYKD:
0:000> .chain
Extension DLL search Path:
...
Extension DLL chain:
pykd.pyd: image 0.3.3.4, API 1.0.0, built Mon May 14 11:14:43 2018
[path: C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\winext\pykd.pyd]
Meanwhile I've found a very simple way for reproducing the issue without needing to launch the whole script (using the Windbg prompt):
0:000> !py
Python 2.7.10 (default, May 23 2015, 09:40:32) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> typeInfo("mfc110u!CPtrList")
Traceback (most recent call last):
File "<console>", line 1, in <module>
SymbolException: 'CPtrList' - symbol not found
In addition to ussrhero's answer, there is following extra information:
The result of x /2 *!CPtrList* contains (amongst many others) the following results:
009530c4 <application>!CPtrList::~CPtrList
009530be <application>!CPtrList::CPtrList
... <application>!CPtrList::...
009abc5c <application>!CPtrList::`RTTI Base Class Array'
009abc4c <application>!CPtrList::`RTTI Class Hierarchy Descriptor'
009bcd18 <application>!CPtrList `RTTI Type Descriptor'
009abc30 <application>!CPtrList::`RTTI Base Class Descriptor at (0,-1,0,64)'
7464e9cb mfc110u!CPtrList::~CPtrList
74544a04 mfc110u!CPtrList::CPtrList
... mfc110u!CPtrList::...
745293b8 mfc110u!CPtrList::`vftable'
747510da mfc110u!CPtrList::`vector deleting destructor'
745293cc mfc110u!CPtrList::`RTTI Complete Object Locator'
7452940c mfc110u!CPtrList::`RTTI Base Class Array'
745293fc mfc110u!CPtrList::`RTTI Class Hierarchy Descriptor'
74795778 mfc110u!CPtrList `RTTI Type Descriptor'
745293e0 mfc110u!CPtrList::`RTTI Base Class Descriptor at (0,-1,0,64)'
746fdc68 mfc110u!CPtrList::classCPtrList
The script I'm using (heap_stat.py) browses through the results of !heap -h 0 and searches for the entry, corresponding with mfc110u!CPtrList::``vtable'.
The result of dt CPtrList starts with the following:
0:000> dt CPtrList
<application>!CPtrList => in other words, no 'mfcu110' entry
+0x000 __VFN_table : Ptr32
I'm already wondering for a long time, what's the difference between the mfc110u!CPtrList and the <application>!CPtrList entries and what's the exact role of the vtable entry in x /2 result?
Any ideas?
Thanks
Meanwhile I've found the solution:
Apparently for some objects, the module prefix needs to be removed:
>>> typeInfo("mdf110u!CPtrList")
-> SymbolException
>>> typeInfo("CPtrList")
-> this is working!!!
Try to see how WinDBG locates this type:
dt CPtrList
It maybe mfc110u does not contain type information fot CPtrList
Example code:
import pycuda.autoinit
import pycuda.driver as drv
import numpy
from pycuda.compiler import SourceModule
mod = SourceModule("""
__global__ void multiply_them(float *dest, float *a, float *b)
{
const int i = threadIdx.x;
dest[i] = a[i] * b[i];
}
""")
multiply_them = mod.get_function("multiply_them")
a = numpy.random.randn(400).astype(numpy.float32)
b = numpy.random.randn(400).astype(numpy.float32)
dest = numpy.zeros_like(a)
multiply_them(
drv.Out(dest), drv.In(a), drv.In(b),
block=(400,1,1), grid=(1,1))
print dest-a*b
Results:
Traceback (most recent call last):
File "test.py", line 12, in <module>
""")
File "build/bdist.linux-x86_64/egg/pycuda/compiler.py", line 238, in __init__
File "build/bdist.linux-x86_64/egg/pycuda/compiler.py", line 223, in compile
File "build/bdist.linux-x86_64/egg/pycuda/compiler.py", line 149, in _find_pycuda_include_path
ImportError: No module named pycuda
Sounds simple enough, so lets test this.
Python 2.7.1 (r271:86832, Feb 17 2011, 14:13:40)
[GCC 4.3.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pycuda
>>> pycuda
<module 'pycuda' from '/home/abolster/lib/python2.7/site-packages/pycuda-0.94.2-py2.7-linux-x86_64.egg/pycuda/__init__.pyc'>
>>>
Ok, thats weird...
Long story short, even stepping through the file line by line into the python console, nothing goes wrong until the actual execution of the mod=SourceModule() line.
(Final Traceback, I promise)
/home/abolster/lib/python2.7/site-packages/pycuda-0.94.2-py2.7-linux-x86_64.egg/pycuda/compiler.pyc in _find_pycuda_include_path()
147 def _find_pycuda_include_path():
148 from imp import find_module
--> 149 file, pathname, descr = find_module("pycuda")
150
151 # Who knew Python installation is so uniform and predictable?
ImportError: No module named pycuda
So it looks like pycuda is getting different include dirs than runtime python, which shouldn't happen (as i understand it)
Any ideas? (Sorry for the long question)
Talonmies borought up a point about nvcc not being found; unless python is getting its envars from somewhere I can't think of, there's no reason it shouldn't :
[bolster#dellgpu src]$ which nvcc
~/cuda/bin/nvcc
Changing to Python 2.6 and reinstalling relevant modules fixed the problem for the OP.
There is nothing wrong with the code you are trying to run - it should work. My guess is that nvcc cannot be found. Make sure that the path to the nvcc executable is set in your environment before you try using pycuda.compiler.
I think you did not install the CUDA toolkit from nvidia and added the
/usr/local/cuda/lib/
to
LD_LIBRARY_PATH
find the the .so of the pycuda module and give us the output of:
>lld pycuda.so
I'm using ctypes to load a DLL in Python. This works great.
Now we'd like to be able to reload that DLL at runtime.
The straightforward approach would seem to be:
1. Unload DLL
2. Load DLL
Unfortunately I'm not sure what the correct way to unload the DLL is.
_ctypes.FreeLibrary is available, but private.
Is there some other way to unload the DLL?
you should be able to do it by disposing the object
mydll = ctypes.CDLL('...')
del mydll
mydll = ctypes.CDLL('...')
EDIT: Hop's comment is right, this unbinds the name, but garbage collection doesn't happen that quickly, in fact I even doubt it even releases the loaded library.
Ctypes doesn't seem to provide a clean way to release resources, it does only provide a _handle field to the dlopen handle...
So the only way I see, a really, really non-clean way, is to system dependently dlclose the handle, but it is very very unclean, as moreover ctypes keeps internally references to this handle. So unloading takes something of the form:
mydll = ctypes.CDLL('./mylib.so')
handle = mydll._handle
del mydll
while isLoaded('./mylib.so'):
dlclose(handle)
It's so unclean that I only checked it works using:
def isLoaded(lib):
libp = os.path.abspath(lib)
ret = os.system("lsof -p %d | grep %s > /dev/null" % (os.getpid(), libp))
return (ret == 0)
def dlclose(handle)
libdl = ctypes.CDLL("libdl.so")
libdl.dlclose(handle)
It is helpful to be able to unload the DLL so that you can rebuild the DLL without having to restart the session if you are using iPython or similar work flow. Working in windows I have only attempted to work with the windows DLL related methods.
REBUILD = True
if REBUILD:
from subprocess import call
call('g++ -c -DBUILDING_EXAMPLE_DLL test.cpp')
call('g++ -shared -o test.dll test.o -Wl,--out-implib,test.a')
import ctypes
import numpy
# Simplest way to load the DLL
mydll = ctypes.cdll.LoadLibrary('test.dll')
# Call a function in the DLL
print mydll.test(10)
# Unload the DLL so that it can be rebuilt
libHandle = mydll._handle
del mydll
ctypes.windll.kernel32.FreeLibrary(libHandle)
I don't know much of the internals so I'm not really sure how clean this is. I think that deleting mydll releases the Python resources and the FreeLibrary call tells windows to free it. I had assumed that freeing with FreeLibary first would have produced problems so I saved a copy of the library handle and freed it in the order shown in the example.
I based this method on ctypes unload dll which loaded the handle explicitly up front. The loading convention however does not work as cleanly as the simple "ctypes.cdll.LoadLibrary('test.dll')" so I opted for the method shown.
windows and linux compatible minimal reproducible example from 2020
overview of similar discussion
Here an overview of similar discussions (where I constructed this answer from).
How can I unload a DLL using ctypes in Python?
ctypes unload dll
Unload shared library inside ctypes loaded shared library
forcing ctypes.cdll.LoadLibrary() to reload library from file
minimal reproducible example
This is for windows and linux, hence there are 2 scripts given for compilation.
Tested under:
Win 8.1, Python 3.8.3 (anaconda), ctypes 1.1.0, mingw-w64 x86_64-8.1.0-posix-seh-rt_v6-rev0
Linux Fedora 32, Python 3.7.6 (anaconda), ctypes 1.1.0, g++ 10.2.1
cpp_code.cpp
extern "C" int my_fct(int n)
{
int factor = 10;
return factor * n;
}
compile-linux.sh
#!/bin/bash
g++ cpp_code.cpp -shared -o myso.so
compile-windows.cmd
set gpp="C:\Program Files\mingw-w64\x86_64-8.1.0-posix-seh-rt_v6-rev0\mingw64\bin\g++.exe"
%gpp% cpp_code.cpp -shared -o mydll.dll
PAUSE
Python code
from sys import platform
import ctypes
if platform == "linux" or platform == "linux2":
# https://stackoverflow.com/a/50986803/7128154
# https://stackoverflow.com/a/52223168/7128154
dlclose_func = ctypes.cdll.LoadLibrary('').dlclose
dlclose_func.argtypes = [ctypes.c_void_p]
fn_lib = './myso.so'
ctypes_lib = ctypes.cdll.LoadLibrary(fn_lib)
handle = ctypes_lib._handle
valIn = 42
valOut = ctypes_lib.my_fct(valIn)
print(valIn, valOut)
del ctypes_lib
dlclose_func(handle)
elif platform == "win32": # Windows
# https://stackoverflow.com/a/13129176/7128154
# https://stackoverflow.com/questions/359498/how-can-i-unload-a-dll-using-ctypes-in-python
lib = ctypes.WinDLL('./mydll.dll')
libHandle = lib._handle
# do stuff with lib in the usual way
valIn = 42
valOut = lib.my_fct(valIn)
print(valIn, valOut)
del lib
ctypes.windll.kernel32.FreeLibrary(libHandle)
A more general solution (object oriented for shared libraries with dependencies)
If a shared library has dependencies, this does not necessarily work anymore (but it can - depends on the dependency ^^). I did not investigate the very details, but it looks like the mechanism is the following: library and dependency are loaded. As the dependency is not unloaded, the library can not get unloaded.
I found, that if I include OpenCv (Version 4.2) into my shared library, this messes up the unloading procedure. The following example was only tested on the linux system:
code.cpp
#include <opencv2/core/core.hpp>
#include <iostream>
extern "C" int my_fct(int n)
{
cv::Mat1b mat = cv::Mat1b(10,8,(unsigned char) 1 ); // change 1 to test unloading
return mat(0,1) * n;
}
Compile with
g++ code.cpp -shared -fPIC -Wall -std=c++17 -I/usr/include/opencv4 -lopencv_core -o so_opencv.so
Python code
from sys import platform
import ctypes
class CtypesLib:
def __init__(self, fp_lib, dependencies=[]):
self._dependencies = [CtypesLib(fp_dep) for fp_dep in dependencies]
if platform == "linux" or platform == "linux2": # Linux
self._dlclose_func = ctypes.cdll.LoadLibrary('').dlclose
self._dlclose_func.argtypes = [ctypes.c_void_p]
self._ctypes_lib = ctypes.cdll.LoadLibrary(fp_lib)
elif platform == "win32": # Windows
self._ctypes_lib = ctypes.WinDLL(fp_lib)
self._handle = self._ctypes_lib._handle
def __getattr__(self, attr):
return self._ctypes_lib.__getattr__(attr)
def __del__(self):
for dep in self._dependencies:
del dep
del self._ctypes_lib
if platform == "linux" or platform == "linux2": # Linux
self._dlclose_func(self._handle)
elif platform == "win32": # Windows
ctypes.windll.kernel32.FreeLibrary(self._handle)
fp_lib = './so_opencv.so'
ctypes_lib = CtypesLib(fp_lib, ['/usr/lib64/libopencv_core.so'])
valIn = 1
ctypes_lib.my_fct.argtypes = [ctypes.c_int]
ctypes_lib.my_fct.restype = ctypes.c_int
valOut = ctypes_lib.my_fct(valIn)
print(valIn, valOut)
del ctypes_lib
Let me know, when there are any issues with the code examples or the explanation given so far. Also if you know a better way! It would be great, if we could settle the issue once and for all.
For total cross-compatibility: I maintain a list of various dlclose() equivalents for each platform and which library to get them from. It's a bit of a long list but feel free to just copy/paste it.
import sys
import ctypes
import platform
OS = platform.system()
if OS == "Windows": # pragma: Windows
dll_close = ctypes.windll.kernel32.FreeLibrary
elif OS == "Darwin":
try:
try:
# macOS 11 (Big Sur). Possibly also later macOS 10s.
stdlib = ctypes.CDLL("libc.dylib")
except OSError:
stdlib = ctypes.CDLL("libSystem")
except OSError:
# Older macOSs. Not only is the name inconsistent but it's
# not even in PATH.
stdlib = ctypes.CDLL("/usr/lib/system/libsystem_c.dylib")
dll_close = stdlib.dlclose
elif OS == "Linux":
try:
stdlib = ctypes.CDLL("")
except OSError:
# Alpine Linux.
stdlib = ctypes.CDLL("libc.so")
dll_close = stdlib.dlclose
elif sys.platform == "msys":
# msys can also use `ctypes.CDLL("kernel32.dll").FreeLibrary()`. Not sure
# if or what the difference is.
stdlib = ctypes.CDLL("msys-2.0.dll")
dll_close = stdlib.dlclose
elif sys.platform == "cygwin":
stdlib = ctypes.CDLL("cygwin1.dll")
dll_close = stdlib.dlclose
elif OS == "FreeBSD":
# FreeBSD uses `/usr/lib/libc.so.7` where `7` is another version number.
# It is not in PATH but using its name instead of its path is somehow the
# only way to open it. The name must include the .so.7 suffix.
stdlib = ctypes.CDLL("libc.so.7")
dll_close = stdlib.close
else:
raise NotImplementedError("Unknown platform.")
dll_close.argtypes = [ctypes.c_void_p]
You can then use dll_close(dll._handle) to unload a library dll = ctypes.CDLL("your-library").
This list is taken from this file. I will update the master branch every time I encounter a new platform.
Piotr's answer helped me, but I did run into one issue on 64-bit Windows:
Traceback (most recent call last):
...
ctypes.ArgumentError: argument 1: <class 'OverflowError'>: int too long to convert
Adjusting the argument type of the FreeLibrary call as suggested in this answer solved this for me.
Thus we arrive at the following complete solution:
import ctypes, ctypes.windll
def free_library(handle):
kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
kernel32.FreeLibrary.argtypes = [ctypes.wintypes.HMODULE]
kernel32.FreeLibrary(handle)
Usage:
lib = ctypes.CDLL("foobar.dll")
free_library(lib._handle)
If you need this functionality, you could write 2 dlls where dll_A loads/Unloads library from dll_B. Use dll_A as as python interface-loader and passthrough for functions in dll_B.