I am currently learning how to write Python (v3.5) extension modules using the Python C API. Some operations, like fast numerical work, are best done in C, while other operations, like string manipulation, are far easier to implement in Python. Is there an agreed-upon way to use both Python and C code to define a new type?
For example, I've written a Matrix type in C that supports basic storage and arithmetic operations. I want to define the Matrix.__str__ using Python, where string manipulations are much easier and I don't need to worry about cstrings.
I attempted to define the __str__ method when the module loads in __init__.py as follows:
from mymodule._mymodule import Matrix;
def as_str(self):
print("This is a matrix!");
Matrix.__str__ = as_str;
When I run this code, I get a TypeError: can't set attributes of built-in/extension type 'matey.Matrix'. Is there an acceptable way to do this? If the solution is to subclass Matrix instead, what is the best way to keep my C base classes / Python subclasses organized within a module?
Personally, I wouldn't try and do object-oriented stuff in C. I'd stick to writing a module which exposes some (stateless) functions.
If I wanted the Python interface to be object oriented, I'd write a class in Python which imports that (C extension) module and consumes functions from it. Maintaining of any state would all be done in Python.
You could instead define a _Matrix type which you then extend with a traditional OOP approach
from mymodule._mymodule import _Matrix;
class Matrix(_Matrix):
def __str__(self):
return "This is a matrix!"
Related
I'm working with Tensorflow in Python. In a custom written function I found #tf_export() before the function definition like below, the function of which I don't understand. Could somebody explain?
#tf_export("signal.ifftshift")
def ifftshift(x, axes=None, name=None):
As I understand, it allows Tensorflow to expose a function or class under a different name. For example, the Server class within the distribute module actually lives in the training/server_lib.py file within the repo, but, since it is exported as distribute.Server, you can use it like tf.distribute.Server().
# training/server_lib.py
#tf_export("distribute.Server", v1=["distribute.Server", "train.Server"])
#deprecation.deprecated_endpoints("train.Server")
class Server(object):
...
It makes it confusing to find the code, but I imagine it's a more flexible way to create these "logical" modules.
It is a convenient way to output dot delimited symbols directly to the tf API. Namely, a user can access ifftshift() with tf.signal.ifftshift(), without caring about the true path (here tf.python.ops.signal.fft_ops.ifftshif()).
I've seen somewhere that there was a way to change some object functions in python
def decorable(cls):
cls.__lshift__ = lambda objet, fonction: fonction(objet)
return cls
I wondered if you could do things like in ruby, with the :
number.times
Can we actually change some predefined classes by applying the function above to the class int for example? If so, any ideas how I could manage to do it? And could you link me the doc of python showing every function (like lshift) that can be changed?
Ordinarily not -
as a rule, Python types defined in native code -in CPython can't be monkey patched to have new methods. Although there are means to do that with direct memory access and changing the C object structures, using CPython - that is not considered "clever", "beautiful", much less usable. (check https://github.com/clarete/forbiddenfruit)
That said, for class hierarchies you define on your own packages, that pretty much works - any magic "dunder" method that is set changes the behavior for all objects of that class, in all the process.
So, you can't do that to Python's "int" - but you can have a
class MyInt(int):
pass
a = MyInt(10)
MyInt.__rshift__ = lambda self, other: MyInt(str(self) + str(other))
print(a >> 20)
Will result in 1020 being printed.
The Python document thta tells about all the magic methods taht are used by the language is the Data Model:
https://docs.python.org/3/reference/datamodel.html
In module source to create new classes I must use next declarative style constructions:
class_<MyClass>("MyClass")
.def("my_method", &MyClass::my_method)
.def("my_second_method", &MyClass::my_second_method);
But if I need to create classes at run-time? For example module function will be return new class. What I must do?
In fact I need create new types on the fly. In my case this typed fixed dictonairies and typed arrays. I need this for optimize my existing code: overhead problem. In my project used data types that transmitted via network. Its a reason to create fixed dict classes in runtime (every class will store individual fields with specified names, like a structs in C++, it's a fixed dict and typed arrays, which the holds element type and array of data of this type).
This will be looks something like this in python code:
from MyCPPLib import DeclareFixedDictonary, DeclareTypedArray
# new used defined data type
NewClass = DeclareFixedDictonary([('field_1', int32), ('field_2', String])
# instance of this type
new_instance = NewClass(4, "Hi")
new_instance['field_1'] = 6
new_instance['field_2'] = "qweqwe"
# ----------------------------------------------
NewArrayClass = DeclareTypedArray(NewClass)
new_array_instance = NewArrayClass([new_instance, NewClass()])
# ----------------------------------------------
NewComplexClass = DeclareFixedDictonary([('f1', float), ('f2', NewArrayClass), (f3, NewClass)])
# ... etc ...
I think if I will create this features with C++ using Python::Boost or/and Python API then I will get maximum speed-up with my types.
My problem is creating new classes in runtime (by other function, in example this DeclareFixedDictonary and DeclareTypedArray).
Following the docs to declare new python class with Boost I must do something like this:
BOOST_PYTHON_MODULE(DataTypes)
{
class_<DataTypesClass>("DataTypesClass")
.def("set", &DataTypesClass::set)
.def("get", &DataTypesClass::get)
.def("set_item", &DataTypesClass::SetItem)
.def("get_item", &DataTypesClass::GetItem)
;
}
But this is module classes, they can be created only in my module, and I can't use class_ in another module function, it's gives error. Maybe Boost has API to create new classes in run-time? Maybe type instances and dynamically filling attributes. Or maybe best way to do this is code generation? But is more hard than reflection system in Python.
I have a file that I read from which has definitions of ctypes that are used in a separate project. I can read the file and obtain all the necessary information to create a ctype that I want in Python like the name, fields, bitfields, ctype base class (Structure, Union, Enum, etc), and pack.
I want to be able to create a ctype class from the information above. I also want these ctypes to be pickleable.
I currently have two solutions, both of which I feel like are hacks.
Solution 1
Generate a Python code object in an appropriate ctype format by hand or with the use of something like Jinja2 and then evaluate the python code object.
This solution has the downside of using eval. I always try to stay away from eval and I don't feel like this is a good place to use it.
Solution 2
Create a ctype dynamically in a function like so:
from ctypes import Structure
def create_ctype_class(name, base, fields, pack):
class CtypesStruct(base):
_fields_ = fields
_pack_ = pack
CtypesStruct.__name__ = name
return CtypesStruct
ctype = create_ctype_class('ctype_struct_name', ctypes.Structure,
[('field1', ctypes.c_uint8)], 4)
This solution isn't so bad, but setting the name of the class is ugly and the type cannot be pickled.
Is there a better way of creating a dynamic ctype class?
Note: I am using Python 2.7
Solution 2 is probably your better option, though if you're also writing such classes statically, you may want to use a metaclass to deduplicate some of that code. If you need your objects to be pickleable, then you'll need a way to reconstruct them from pickleable objects. Once you've implemented such a mechanism, you can make the pickle module aware of it with a __reduce__() method.
I would go with a variant of Solution 1. Instead of evaling code, create a directory with an __init__.py (i.e. a package), add it to your sys.path and write out an entire python module containing all of the classes. Then you can import them from a stable namespace which will make pickle happier.
You can either take the output and add it to your app's source code or dynamically recreate it and cache it on a target machine at runtime.
pywin32 uses an approach like this for caching classes generated from ActiveX interfaces.
I've built a module in Python in one single file without using classes. I do this so that using some api module becomes easier. Basically like this:
the_module.py
from some_api_module import some_api_call, another_api_call
def method_one(a, b):
return some_api_call(a + b)
def method_two(c, d, e):
return another_api_call(c * d * e)
I now need to built many similar modules, for different api modules, but I want all of them to have the same basic set of methods so that I can import any of these modules and call a function knowing that this function will behave the same in all the modules I built. To ensure they are all the same, I want to use some kind of abstract base module to build upon. I would normally grab the Abstract Base Classes module, but since I don't use classes at all, this doesn't work.
Does anybody know how I can implement an abstract base module on which I can build several other modules without using classes? All tips are welcome!
You are not using classes, but you could easily rewrite your code to do so.
A class is basically a namespace which contains functions and variables, as is a module.
Should not make a huge difference whether you call mymodule.method_one() or mymodule.myclass.method_one().
In python there is no such thing as interfaces which you might know from java.
The paradigm in python is Duck typing, that means more or less that for a given module you can tell whether it implements your API if it provides the right methods.
Python does this i.e. to determine what to do if you call myobject[i] on an instance of your class myclass. It looks whether the class has the method __getitem__ and if it does so, it replaces myobject[i] by myobject.__getitem__(i).
Yout don't have to tell python that your class supports this kind of access, python just figures it out from the way you defined your class.
The same way you should determine whether your module implements your API.
Maybe you want to look inside the hidden dictionary mymodule.__dict__ after import mymodulewhich contains all function names and pointers to them of your module. You could then check whether the right functions are present and raise an error otherwise
import my_module_4
#check if my_module_4 implements api
if all(func in my_module_4.__dict__ for func in ("method_one","method_two"):
print "API implemented"
else:
print "Warning: Not all API functions found in my_module_4"