Build an extension class by inheriting a python class in Cython

Question

I would like to build a child class by inheriting a python class in Cython. It seems I cannot do it directly, as I got an error below. Is there any workaround for it?

Code (osmium is a third-party python package, which can be installed using pip):

import osmium

cdef class CounterHandler(osmium.SimpleHandler):
    cdef list nodes, ways, relations
    def __init__(self):
        osmium.SimpleHandler.__init__(self)
        self.nodes = []
        self.ways = []
        self.relations = []

    def node(self, n):
        pass

    def way(self, w):
        pass

    def relation(self, r):
        pass

Error message:

add.pyx:22:32: First base of 'CounterHandler' is not an extension type
Traceback (most recent call last):
  File "setup.py", line 11, in <module>
    ext_modules=cythonize("add.pyx"))
  File "C:\ProgramData\Miniconda3\envs\osmium\lib\site-packages\Cython\Build\Dependencies.py", line 1102, in cythonize
    cythonize_one(*args)
  File "C:\ProgramData\Miniconda3\envs\osmium\lib\site-packages\Cython\Build\Dependencies.py", line 1225, in cythonize_one
    raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: add.pyx

I tried the Solutions provided by DavidW

Solution 2 Code:

import osmium

cdef class CounterHandlerBase:
    cdef list nodes, ways, relations
    def __init__(self):
        self.nodes = []
        self.ways = []
        self.relations = []

    cdef node(self, n):
        pass

    cdef way(self, w):
        pass

    cdef relation(self, r):
        pass

class CounterHandler(CounterHandlerBase, osmium.SimpleHandler):    # osmium.SimpleHandler
    def __init__(self):
        CounterHandlerBase.__init__(self)
        osmium.SimpleHandler.__init__(self)

Error Message:

Traceback (most recent call last):
  File "C:/Users/Administrator/Dropbox (ASU)/Work/CAVLite/OSM2GMNS/V2/cython_test/tets.py", line 7, in <module>
    import solution2 as solution
  File "solution2.pyx", line 28, in init solution2
    class CounterHandler(CounterHandlerBase, osmium.SimpleHandler):    # osmium.SimpleHandler
TypeError: multiple bases have instance lay-out conflict

Solution 3 Code:

import osmium

cdef class DummyBase:
    def __init__(self):
        pass

cdef class CounterHandler(DummyBase, osmium.SimpleHandler):    # osmium.SimpleHandler
    cdef list nodes, ways, relations
    def __init__(self):
        DummyBase.__init__(self)
        osmium.SimpleHandler.__init__(self)
        self.nodes = []
        self.ways = []
        self.relations = []

    cdef node(self, n):
        pass

    cdef way(self, w):
        pass

    cdef relation(self, r):
        pass

Error Message:

Traceback (most recent call last):
  File "C:/Users/Administrator/Dropbox (ASU)/Work/CAVLite/OSM2GMNS/V2/cython_test/tets.py", line 7, in <module>
    import solution3 as solution
  File "solution3.pyx", line 16, in init solution3
    cdef class CounterHandler(DummyBase, osmium.SimpleHandler):    # osmium.SimpleHandler
TypeError: best base 'osmium._osmium.SimpleHandler' must be equal to first base 'solution3.DummyBase'

Answer 1

There's a number of options here:

1. Do you actually need it to be a cdef class? Have you got a real reason for this (beyond a generic, untested belief that "cdef classes are faster")? Maybe you can use a regular class instead? You don't look to be using any attributes that can't be represented in Python (e.g. C pointers). Remember that Cython still compiles def functions of regular classes, so there may not be the speed difference that you imagine.

2. Split it into the bits that need to be a cdef class and the bits that don't (this only works if the interaction with osmium.SimpleHandler is in the bits that don't):

   cdef class CounterHandlerBase:
       # code goes here
   
   class CounterHandler(CounterHandlerBase, osmium.SimpleHandler):
       # more code goes here
   
   

3. The restriction is that the first base must be a cdef class (which is actually a fairly strong restriction that's built into Python). Second/subsequent bases can be regular classes. Therefore you could create a "dummy" cdef base class just to fill that role:

   cdef class DummyBase:
       pass
   
   cdef class CounterHandler(DummyBase, osmium.SimpleHandler):
       # code goes here...
   

---

Edit: Based on the errors you report it looks like osmium.SimpleHandler is already an extension type written in C/C++. Unfortunately this means that is won't be possible to inherit from it in a cdef class because of restrictions on object layout that are built into Python (it's possible that defining it as an "external cdef class" might work, but it looks to be generated from by pybind11 which makes it quite hard work work out the underlying struct).

Therefore, options 2 and 3 are never going to work in this case. Since it's written in C++ already I doubt that re-writing stuff in Cython is going to speed anything up.