We Keep Coding

Defining a Python class in both Python and C

I am currently learning how to write Python (v3.5) extension modules using the Python C API. Some operations, like fast numerical work, are best done in C, while other operations, like string manipulation, are far easier to implement in Python. Is there an agreed-upon way to use both Python and C code to define a new type?
For example, I've written a Matrix type in C that supports basic storage and arithmetic operations. I want to define the Matrix.__str__ using Python, where string manipulations are much easier and I don't need to worry about cstrings.
I attempted to define the __str__ method when the module loads in __init__.py as follows:
from mymodule._mymodule import Matrix;
def as_str(self):
print("This is a matrix!");
Matrix.__str__ = as_str;
When I run this code, I get a TypeError: can't set attributes of built-in/extension type 'matey.Matrix'. Is there an acceptable way to do this? If the solution is to subclass Matrix instead, what is the best way to keep my C base classes / Python subclasses organized within a module?

Personally, I wouldn't try and do object-oriented stuff in C. I'd stick to writing a module which exposes some (stateless) functions.
If I wanted the Python interface to be object oriented, I'd write a class in Python which imports that (C extension) module and consumes functions from it. Maintaining of any state would all be done in Python.

You could instead define a _Matrix type which you then extend with a traditional OOP approach
from mymodule._mymodule import _Matrix;
class Matrix(_Matrix):
def __str__(self):
return "This is a matrix!"

Cython: a class needs to have as its attribute a python list of Cython extension types -- thus, this class cannot be an extension type itself?

Let us say I have a Cython extension type named Point.
Then, I need to create a class called Points, which has as one of its attributes, a Python list of Point objects. Now, Python lists, as far as I understand, cannot be attributes of an extension type, as only C data types can be.
I want to use Python lists to hold the list of Cython extension types, because I have heard that its the easiest way to do so, and access of Python lists through Cython is quit efficient.
Thus, Points has to be a normal Python/Cython class, not an extension type, correct? That way, I can do the following:
def class Points:
def __init__(self, num_points):
self.points = [Point() for x in range(num_points)]
Do I understand correctly?

No, there is no such limitation, Cython extension types can have arbitrary Python objects as attributes. For example you could declare your Points class in the following way:
cdef class Points:
cdef list points
def __init__(self, num_points):
self.points = [Point() for x in range(num_points)]
Note that you need to declare all attributes of Cython extension types in advance. If you do not want to restrict the attribute to a specific Python object you can also use object instead of list. If you want to expose your points attribute to Python, i.e. allow direct access from Python, you need to declare it as public (i.e. cdef public list points).
Have a look at the documentation about attributes of extension types for more details and the second example of the properties section also provides an example how an extension type can wrap a list without providing direct access.

I don't know about any such restriction with Cython extension types. But, from the standard documentation on general extension types, it's clear you can create arbitrary PyObjects as member variables in an extension type (see the Noddy2 example with unrestricted PyObject members first and last which are later refined to be restricted to string types). If you went this route, you could expose a member as a PyObject and pass it a list by convention or fully restrict it to be a list.

Dynamically create Ctypes in Python

I have a file that I read from which has definitions of ctypes that are used in a separate project. I can read the file and obtain all the necessary information to create a ctype that I want in Python like the name, fields, bitfields, ctype base class (Structure, Union, Enum, etc), and pack.
I want to be able to create a ctype class from the information above. I also want these ctypes to be pickleable.
I currently have two solutions, both of which I feel like are hacks.
Solution 1
Generate a Python code object in an appropriate ctype format by hand or with the use of something like Jinja2 and then evaluate the python code object.
This solution has the downside of using eval. I always try to stay away from eval and I don't feel like this is a good place to use it.
Solution 2
Create a ctype dynamically in a function like so:
from ctypes import Structure
def create_ctype_class(name, base, fields, pack):
class CtypesStruct(base):
_fields_ = fields
_pack_ = pack
CtypesStruct.__name__ = name
return CtypesStruct
ctype = create_ctype_class('ctype_struct_name', ctypes.Structure,
[('field1', ctypes.c_uint8)], 4)
This solution isn't so bad, but setting the name of the class is ugly and the type cannot be pickled.
Is there a better way of creating a dynamic ctype class?
Note: I am using Python 2.7

Solution 2 is probably your better option, though if you're also writing such classes statically, you may want to use a metaclass to deduplicate some of that code. If you need your objects to be pickleable, then you'll need a way to reconstruct them from pickleable objects. Once you've implemented such a mechanism, you can make the pickle module aware of it with a __reduce__() method.

I would go with a variant of Solution 1. Instead of evaling code, create a directory with an __init__.py (i.e. a package), add it to your sys.path and write out an entire python module containing all of the classes. Then you can import them from a stable namespace which will make pickle happier.
You can either take the output and add it to your app's source code or dynamically recreate it and cache it on a target machine at runtime.
pywin32 uses an approach like this for caching classes generated from ActiveX interfaces.

Dynamic Python Class Definition in SQLAlchemy

I'm creating a backend application with SQLAlchemy using the declarative base. The ORM requires about 15 tables each of which maps to a class object in SQLAlchemy. Because these class objects are all defined identically I thought a factory pattern could produce the classes more concisely. However, these classes not only have to be defined, they have to be assigned to unique variable names so they can be imported and used through the project.
(Sorry if this question is a bit long, I updated it as I better understood the problem.)
Because we have so many columns (~1000) we define their names and types in external text files to keep things readable. Having done that one way to go about declaring our models is like this:
class Foo1(Base):
__tablename___ = 'foo1'
class Foo2(Base):
__tablename___ = 'foo2'
... etc
and then I can add the columns by looping over the contents of the external text file and using the setattr() on each class definition.
This is OK but it feels too repetitive as we have about 15 tables. So instead I took a stab at writing a factory function that could define the classes dynamically.
def orm_factory(class_name):
class NewClass(Base):
__tablename__ = class_name.lower()
NewClass.__name__ = class_name.upper()
return NewClass
Again I can just loop over the columns and use setattr(). When I put it together it looks like this:
for class_name in class_name_list:
ORMClass = orm_factory(class_name)
header_keyword_list = get_header_keyword_list(class_name)
define_columns(ORMClass, header_keyword_list)
Where get_header_keyword_list gets the column information and define_columns performs the setattr() assignment. When I use this and run Base.metadata.create_all() the SQL schema get generated just fine.
But, when I then try to import these class definitions into another model I get an error like this:
SAWarning: The classname 'NewClass' is already in the registry of this declarative base, mapped to <class 'ql_database_interface.IR_FLT_0'>
This, I now realize makes total sense based on what I learned yesterday: Python class variable name vs __name__.
You can address this by using type as a class generator in your factory function (as two of the answers below do). However, this does not solve the issue of being able to import the class because the while the classes are dynamically constructed in the factory function the variable the output of that function is assigned to is static. Even if it were dynamic, such as a dictionary key, it has to be in the module name space in order to be imported from another module. See my answer for more details.

This sounds like a sketchy idea. But it's fun to solve so here is how you make it work.
As I understand it, your problem is you want to add dynamically created classes to a module. I created a hack using a module and the init.py file.
dynamicModule/__init__.py:
import dynamic
class_names = ["One", "Two", "Three"]
for new_name in class_names:
dynamic.__dict__['Class%s' % new_name] = type("Class%s" % (new_name), (object,), {'attribute_one': 'blah'})
dynamicModule/dynamic.py:
"""Empty file"""
test.py:
import dynamicModule
from dynamicModule import dynamic
from dynamicModule.dynamic import ClassOne
dynamic.ClassOne
"""This all seems evil but it works for me on python 2.6.5"""
__init__.py:
"""Empty file"""

[Note, this is the original poster]
So after some thinking and talking to people I've decided that that ability to dynamically create and assign variables to class objects in the global name space in this way this just isn't something Python supports (and likely with good reason). Even though I think my use case isn't too crazy (pumping out predefined list of identically constructed classes) it's just not supported.
There are lots of questions that point towards using a dictionary in a case like this, such as this one: https://stackoverflow.com/a/10963883/1216837. I thought of something like that but the issue is that I need those classes in the module name space so I can import them into other modules. However, adding them with globals() like globals()['MyClass'] = class_dict['MyClass'] seems like it's getting pretty out there and my impression is people on SO frown on using globals() like this.
There are hacks such as the one suggested by patjenk but at a certain point the obfuscation and complexity out weight the benefits of the clarity of declaring each class object statically. So while it seems repetitive I'm just going to write out all the class definitions. Really, this end up being pretty concise/maintainable:
Class1 = class_factory('class1')
Class2 = class_factory('class2')
...

How to access with ctypes to functions returning custom types coded in a Delphi dll?

I have a Delhi DLL that is exposing a function with the following signature:
Function MyFunc(ObjID : Cardinal) : TMyRec; stdcall;
where the records are defined so:
type TMyRec = record
Count : Cardinal;
Items : array of TMyItemRec;
end;
type TMyItemRec = record
ID : Cardinal;
Params : array of String;
end;
Now my question is: how can I acces results of MyFunc calling the dll with Python ctypes? I coded two classes that mimic the types
from ctypes import *
class TMyItemRec(Structure):
_fields_ = [("ID", c_int), ("Params", POINTER(c_wchar_p))]
class TMyRec(Structure):
_fields_ = [("Count", c_int), ("Params", POINTER(TMyItemRec))]
but when I try to read data like this:
my_dll = windll.Script
def GetMyRec(ID):
my_dll.MyFunc.argtypes = [c_uint]
my_dll.MyFunc.restype = TClilocRec
return my_dll.Script_GetClilocRec(ID)
I get access violation error.

You cannot pass Delphi managed types like dynamic arrays to non-Delphi code. You cannot expect to call functions with those data types.
You will need to re-design your interface. You need to use simple types and records containing simple types. If you need arrays then you'll have to pass a pointer to the first element, and the length, rather than using Delphi specific managed types. Use the Windows API as your template for how to design interop interfaces.
The other thing you'll need to deal with is that function return values are handled differently in Delphi than in most other Windows compilers. So records that do not fit in a register will need to be passed as var parameters rather than as function return values.