I was wondering whether objects serialized using CPython's cPickle are readable by using IronPython's cPickle; the objects in question do not require any modules outside of the built-ins that both Cpython and IronPython include. Thank you!
If you use the default protocol (0) which is text based, then things should work. I'm not sure what will happen if you use a higher protocol. It's very easy to test this ...
It will work because when you unpickle objects during load() it will use the current definitions of whatever classes you have defined now, not back when the objects were pickled.
IronPython is simply Python with the standard library implemented in C# so that everything emits IL. Both the CPython and the IronPython pickle modules have the same functionality, except one is implemented in C and the other in C#.
Related
I noticed that when my object contains an explicit reference to a module, pickling it will fail because of this.
However, if I stick a reference to a function from that module into my object instead, it can be picked and unpickled successfully.
How come Python can pickle functions, but not modules?
Because they didn't code support for it. C level types (and even modules written in Python are implemented with a C level type) require pickle support to be coded explicitly.
It's not very easy to determine what should be pickled if a module is allowed to be pickled; importing the same name on the other side would seem simple, but if you're actually trying to pickle the module itself, the worry would be that you want to pickle module state as well. It's even more confusing if the module is a C extension module, where module state may not even be exposed to Python itself, only used internally at the C layer.
Given that usually you want specific things from a module, not the whole module (which is usually not referenced as state, just imported at the top level), the benefits of supporting pickling for modules are limited, and the semantics are unclear, they haven't bothered to implement it.
I want to know whether there is a PyMOD(DEINIT)_FUNC?, I know that resources are released when the python script exits, but I would like to make my code as responsible for the memory it uses as possible.
I have searched the documentaion of course, and until now I think there is no function called from python core to the c module when the module is unloaded, but I hope there is and I just don't know how to search.
Python 2 does not support module finalisation, no. See bug 9072:
Please accept that Python indeed does not support unloading modules for severe, fundamental, insurmountable, technical problems, in 2.x.
For Python 3, the C API for module initialisation was overhauled (see PEP 3121) and the new PyModuleDef struct has a m_free slot that takes a callback function; use that to clear out your module memory.
I'm working through some scipy lectures (http://scipy-lectures.github.io/intro/language/standard_library.html#pickle-easy-persistence) and I came across this statement about Pickle:
Useful to store arbitrary objects to a file. Not safe or fast!
What do they mean by this? Not safe (according to Pickle docs) as in don't UnPickle files from an unknown origin or not safe as in you don't always retrieve the original object?
What's the alternative for something safer and faster? I know about cPickle being faster, but I don't think it solves the above definition of safer.
Thanks.
Using pickle in production code is vulnerable by design. Arbitrary code can be executed while unpickling. You can safely unpickle only data from trusted sources. Never unpickle data received from an untrusted or unauthenticated source.
See here for real applications samples.
As for faster alternative, there is marshal, python internal serealization library. But unlike pickle (or cPickle, which is just a C implementation), it is less stable (see docs) and its output being architecture and os independend, depends on python version. That is object marshal'ed on Windows platform with python 2.7.5 is guaranteed to be un-marshalable on OS X or Ubuntu with python 2.7.5 installed, but not guaranteed to be un-marshalable with python 2.6 on Windows.
Another faster, safer by design, but less functional serialization alternative is JSON.
The original module Pickle is almost never used.
If you need to do it fast, use cPickle.
If you need a safe one, try sPickle.
I am trying to use the rubypython gem. Not sure how to call standard python functions like len and set. In the python examples I see len(text3) and set(text3).
How do I call these in rubypython?
Here is the link to rubypython: http://rubypython.rubyforge.org/
Well, my Ruby knowledge is limited, and my knowledge of the rubypython gem is non-existent. However, I do know the standard functions you refer to a part of the __builtin__ module, which is automatically imported into the python namespace. Fortunately, there's nothing preventing you from importing it explicitly again (which is perfectly safe in Python). You then might be able do something like __builtin__.set(). No guarantees, though.
RubyPython::PyMainClass has a public instance method builtin()
You can use that to call the standard functions.
I read through the following two threads on wrapping C library and C++ library, I am not sure I get it yet. The C++ library I am working with does use class and template, but not in any over-sophisticated way. What are issues or caveats of wrapping it with ctypes (besides the point that you can do so in pure python etc)?
PyCXX , Cython and boost::python are three other choices people mentioned, is there any consensus which one is more suitable for C++?
Thanks
Oliver
In defence of boost::python, given Alexander's answer on ctypes:
Boost python provides a very "c++" interface between c++ and python code - even doing things like allowed python subclasses of c++ classes to override virtual methods is relatively straightforward. Here's a potted list of good features:
Allow virtual methods of C++ classes to be overridden by python subclasses.
Bridge between std::vector<>, std::map<> instances and python lists and dictionaries (using vector_indexing_suite and map_indexing_suite)
Automatic sharing of reference counts in smart pointers (boost::shared_ptr, etc) with python reference counts (and you can extend this to any smart pointer).
Fine grained control of ownership when passing arguments and returning values from functions.
Basically, if you have a design where you want to expose a c++ interface in a way faithful to the language, then boost::python is probably the best way to do it.
The only downsides are increased compile time (boost::python makes extensive use of templates), and sometimes opaque error messages if you don't get things quite right.
For C++ a library to be accessible from Python it must use C export names, which basically means that a function named foo will be accessible from ctypes as foo.
This can be achieved only by enclosing the public interface with export C {}, which in turn disallows function overloading and templates therein (only the public interface of the library to be wrapped is relevant, the inner workings are not and may use any C++ features they like).
Reason for this is that C++ compilers use a mechanism called name mangling to generate unique names for overloaded or templated symbols. While ctypes would still find a function provided you knew its mangled name, the mangling scheme depends on the compiler/linker being used and is nothing you can rely on. In short: do not use ctypes to wrap libraries that use C++ features in their public interface.
Cython takes a different approach. It aids you at building a C extension module that does the interfacing with the original library. Therefore, linking to the C++ library is done by the regular C++ linkage mechanism, thus avoiding the aforementioned problem. The trouble with Cython is that C extension libraries need to to be recompiled for every platform, but anyway, this applies to the C++ library to be wrapped as well.
Personally, I'd say that in most cases the time to fire up Cython is a time that is well-spent and will eventually pay off in comparison to ctypes (with an exception for really simple Cish interfaces).
I don't have any experience with boost.python, so I can't comment on it (however, I don't have the impression that it is very popular either).