I want to know whether there is a PyMOD(DEINIT)_FUNC?, I know that resources are released when the python script exits, but I would like to make my code as responsible for the memory it uses as possible.
I have searched the documentaion of course, and until now I think there is no function called from python core to the c module when the module is unloaded, but I hope there is and I just don't know how to search.
Python 2 does not support module finalisation, no. See bug 9072:
Please accept that Python indeed does not support unloading modules for severe, fundamental, insurmountable, technical problems, in 2.x.
For Python 3, the C API for module initialisation was overhauled (see PEP 3121) and the new PyModuleDef struct has a m_free slot that takes a callback function; use that to clear out your module memory.
Related
How exactly can Python call a C library? Tensorflow, for example, I believe is written mostly in C, but can be used from Python. I'm thinking of implementing something like this in my own (interpreted) programming language (written in Go, but I assume it would be a similar process).
What happens when a Python program calls a C function? I'm thinking either RPC or DLLs, but both of them seem unlikely.
cPython has two main ways to call C code: either by loading a shared library and calling its symbols, or by packing C code as Python binary modules and then calling them from Python code as though they were ordinary Python modules, which is how high performance stuff in the standard library is implemented - e.g. json.
Loading a shared library and calling functions from it using the ctypes module is rather trivial, and you can find a lot of examples here: https://docs.python.org/3/library/ctypes.html
Packing your C code as binary Python module requires a lot of boilerplate and careful attention to details such as ref counting, null pointers, etc, and is documented here: https://docs.python.org/2/extending/extending.html
There are several libraries that automate the process and generate binding code for you. One example is boost.python: https://www.boost.org/doc/libs/1_65_0/libs/python/doc/html/tutorial/index.html
I am trying to use the rubypython gem. Not sure how to call standard python functions like len and set. In the python examples I see len(text3) and set(text3).
How do I call these in rubypython?
Here is the link to rubypython: http://rubypython.rubyforge.org/
Well, my Ruby knowledge is limited, and my knowledge of the rubypython gem is non-existent. However, I do know the standard functions you refer to a part of the __builtin__ module, which is automatically imported into the python namespace. Fortunately, there's nothing preventing you from importing it explicitly again (which is perfectly safe in Python). You then might be able do something like __builtin__.set(). No guarantees, though.
RubyPython::PyMainClass has a public instance method builtin()
You can use that to call the standard functions.
How do I, at run-time (no LD_PRELOAD), intercept/hook a C function like fopen() on Linux, a la Detours for Windows? I'd like to do this from Python (hence, I'm assuming that the program is already running a CPython VM) and also reroute to Python code. I'm fine with just hooking shared library functions. I'd also like to do this without having to change the way the program is run.
One idea is to roll my own tool based on ptrace(), or on rewriting code found with dlsym() or in the PLT, and targeting ctypes-generated C-callable functions, but I thought I'd ask here first. Thanks.
You'll find from one of ltrace developer a way to do this. See this post, which includes a full patch in order to catch dynamically loaded library. In order to call it from python, you'll probably need to make a C module.
google-perftools has their own implementation of Detour under src/windows/preamble_patcher* . This is windows-only at the moment, but I don't see any reason it wouldn't work on any x86 machine except for the fact that it uses win32 functions to look up symbol addresses.
A quick scan of the code and I see these win32 functions used, all of which have linux versions:
GetModuleHandle/GetProcAddress : get the function address. dlsym can do this.
VirtualProtect : to allow modification of the assembly. mprotect.
GetCurrentProcess: getpid
FlushInstructionCache (apparently a nop according to the comments)
It doesn't seem too hard to get this compiled and linked into python, but I'd send a message to the perftools devs and see what they think.
I was wondering whether objects serialized using CPython's cPickle are readable by using IronPython's cPickle; the objects in question do not require any modules outside of the built-ins that both Cpython and IronPython include. Thank you!
If you use the default protocol (0) which is text based, then things should work. I'm not sure what will happen if you use a higher protocol. It's very easy to test this ...
It will work because when you unpickle objects during load() it will use the current definitions of whatever classes you have defined now, not back when the objects were pickled.
IronPython is simply Python with the standard library implemented in C# so that everything emits IL. Both the CPython and the IronPython pickle modules have the same functionality, except one is implemented in C and the other in C#.
I have heard that Ctypes can cause crashes (or stop errors) in Python and windows. Should I stay away from their use? Where did I hear? It was back when I tried to control various aspects of windows, automation, that sort of thing.
I hear of swig, but I see Ctypes more often than not. Any danger here? If so, what should I watch out for?
I did search for ctype pro con python.
In terms of robustness, I still think swig is somewhat superior to ctypes, because it's possible to have a C compiler check things more thoroughly for you; however, this is pretty moot by now (while it loomed larger in earlier ctypes versons), thanks to the argtypes feature #Mark already mentioned. However, there is no doubt that the runtime overhead IS much more significant for ctypes than for swig (and sip and boost python and other "wrapping" approaches): so, I think of ctypes as a convenient way to reach for a few functions within a DLL when the calls happen outside of a key bottleneck, not as a way to make large C libraries available to Python in performance-critical situations.
For a nice middle way between the runtime performance of swig (&c) and the convenience of ctypes, with the added bonus of being able to add more code that can use a subset of Python syntax yet run at just about C-code speeds, also consider Cython -- a python-like language that compiles down to C and is specialized for writing Python-callable extensions and wrapping C libraries (including ones that may be available only as static libraries, not DLLs: ctypes wouldn't let you play with those;-).
ctypes is a safe module to use, if you use it right.
Some libraries provide a lower level access to things, some modules simply allow you to shoot yourself in the foot. So naturally some modules are more dangerous than others. This doesn't mean you should not use them though!
You probably heard someone referring to something like this:
#Crash python interpreter
from ctypes import *
def crashme():
c = c_char('x')
p = pointer(c)
i = 0
while True:
p[i] = 'x'
i += 1
The python interpreter crashing is different than just the python code itself erroring out with a runtime error. For example infinite recursion with a default recursion limit set would cause a runtime error but the python interpreter would still be alive afterwards.
Another good example of this is with the sys module. You wouldn't stop using the sys module though because it can crash the python interpreter.
import sys
sys.setrecursionlimit(2**30)
def f(x):
f(x+1)
#This will cause no more resources left and then crash the python interpreter
f(1)
There are many libraries as well that provide lower level access. For example the The gc module can be manipulated to give access to partially constructed object, accessing fields of which can cause crashes.
Reference and ideas taken from: Crashing Python
ctypes can indeed cause crashes, if the C library you're using can already cause crashes.
If anything, ctypes can help reduce crashes, because you can enforce runtime type safety with the argtypes property on C functions using ctypes.
But if your C library is already stable and tested, there is absolutely no reason not to use ctypes if it performs what you need in terms of bringing C and Python together.
I highly suggest you look into reading this book:
Gray Hat Python: Python Programming for Hackers and Reverse Engineers
The book functions as an in-depth tutorial for the ctypes library, and shows you how to run incredibly low-level code