Custom Converter for deserializing nested sealed class with datastore entities

Question

We have a main object model that has a collection of Events on it among other values. These events are a known set of types such as Created, Errored, Completed, etc... each with a handful of overridden fields but some specific fields for that event. To us it makes sense to have these as data classes under a Sealed Type. We're using GCP's Datastore and the com.google.cloud:spring-cloud-gcp-starter-data-datastore library for reading and writing.

The error we're running into occurs on the datastoreTemplate.findById(...) and we believe it has something to do with the Sealed classes for the Events when deserializing the entities. Stacktrace:

Failed to instantiate com.PhoneExample.PhoneEntity$EventEntity using constructor fun <init>(): com.PhoneExample.PhoneEntity.EventEntity with arguments
org.springframework.data.mapping.model.MappingInstantiationException: Failed to instantiate com.PhoneExample.PhoneEntity$EventEntity using constructor fun <init>(): com.PhoneExample.PhoneEntity.EventEntity with arguments 
    at org.springframework.data.mapping.model.ReflectionEntityInstantiator.createInstance(ReflectionEntityInstantiator.java:79)
    at org.springframework.data.mapping.model.ClassGeneratingEntityInstantiator.createInstance(ClassGeneratingEntityInstantiator.java:89)
    at com.google.cloud.spring.data.datastore.core.convert.DefaultDatastoreEntityConverter.read(DefaultDatastoreEntityConverter.java:169)
    at com.google.cloud.spring.data.datastore.core.convert.DefaultDatastoreEntityConverter.read(DefaultDatastoreEntityConverter.java:54)
    at com.google.cloud.spring.data.datastore.core.convert.TwoStepsConversions.convertOnReadSingleEmbedded(TwoStepsConversions.java:205)
    at com.google.cloud.spring.data.datastore.core.convert.TwoStepsConversions.convertOnRead(TwoStepsConversions.java:183)
    at com.google.cloud.spring.data.datastore.core.convert.TwoStepsConversions.convertOnRead(TwoStepsConversions.java:130)
    at com.google.cloud.spring.data.datastore.core.convert.EntityPropertyValueProvider.getPropertyValue(EntityPropertyValueProvider.java:68)
    at com.google.cloud.spring.data.datastore.core.convert.EntityPropertyValueProvider.getPropertyValue(EntityPropertyValueProvider.java:54)
    at com.google.cloud.spring.data.datastore.core.convert.EntityPropertyValueProvider.getPropertyValue(EntityPropertyValueProvider.java:32)
    at org.springframework.data.mapping.model.PersistentEntityParameterValueProvider.getParameterValue(PersistentEntityParameterValueProvider.java:74)
    at org.springframework.data.mapping.model.ClassGeneratingEntityInstantiator.extractInvocationArguments(ClassGeneratingEntityInstantiator.java:276)
    at org.springframework.data.mapping.model.ClassGeneratingEntityInstantiator$EntityInstantiatorAdapter.createInstance(ClassGeneratingEntityInstantiator.java:248)
    at org.springframework.data.mapping.model.ClassGeneratingEntityInstantiator.createInstance(ClassGeneratingEntityInstantiator.java:89)
    at com.google.cloud.spring.data.datastore.core.convert.DefaultDatastoreEntityConverter.read(DefaultDatastoreEntityConverter.java:169)
    at com.google.cloud.spring.data.datastore.core.convert.DefaultDatastoreEntityConverter.read(DefaultDatastoreEntityConverter.java:54)
    at com.google.cloud.spring.data.datastore.core.DatastoreTemplate.convertEntityResolveDescendantsAndReferences(DatastoreTemplate.java:669)
    at com.google.cloud.spring.data.datastore.core.DatastoreTemplate.lambda$convertEntitiesForRead$15(DatastoreTemplate.java:657)
    at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
    at java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1603)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
    at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
    at com.google.cloud.spring.data.datastore.core.DatastoreTemplate.convertEntitiesForRead(DatastoreTemplate.java:659)
    at com.google.cloud.spring.data.datastore.core.DatastoreTemplate.findAllById(DatastoreTemplate.java:264)
    at com.google.cloud.spring.data.datastore.core.DatastoreTemplate.performFindByKey(DatastoreTemplate.java:246)
    at com.google.cloud.spring.data.datastore.core.DatastoreTemplate.findById(DatastoreTemplate.java:140)
    at com.PhoneExample.ExampleTest.processTest(ExampleTest.kt:17)
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.PhoneExample.PhoneEntity$EventEntity]: Is it an abstract class?; nested exception is java.lang.InstantiationException
    at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:215)
    at org.springframework.data.mapping.model.ReflectionEntityInstantiator.createInstance(ReflectionEntityInstantiator.java:77)
Caused by: java.lang.InstantiationException
    at java.base/jdk.internal.reflect.InstantiationExceptionConstructorAccessorImpl.newInstance(InstantiationExceptionConstructorAccessorImpl.java:48)
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
    at kotlin.reflect.jvm.internal.calls.CallerImpl$Constructor.call(CallerImpl.kt:41)
    at kotlin.reflect.jvm.internal.KCallableImpl.call(KCallableImpl.kt:108)
    at kotlin.reflect.jvm.internal.KCallableImpl.callDefaultMethod$kotlin_reflection(KCallableImpl.kt:159)
    at kotlin.reflect.jvm.internal.KCallableImpl.callBy(KCallableImpl.kt:112)
    at org.springframework.beans.BeanUtils$KotlinDelegate.instantiateClass(BeanUtils.java:854)
    at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:196)

I've tried to make the following example as minimal as possible while maintaining the structure we're using.

class ExampleTest(private val datastoreTemplate: DatastoreTemplate) {
    fun processTest() {
        val phone = Phone(
            "123456789",
            ErrorEvent("Error-01", RuntimeException("some exception"), Instant.now())
        )

        val phoneEntity = datastoreTemplate.save(PhoneEntity(phone))
        datastoreTemplate.findById(phoneEntity.id, PhoneEntity::class.java)
    }
}

data class Phone(
    val id: String,
    val events: Event
)

abstract class BaseEvent {
    abstract val name: String
    abstract val time: Instant
}
sealed class Event : BaseEvent()

data class ErrorEvent(
    override val name: String,
    val error: Throwable,
    override val time: Instant
) : Event()


@Entity(name = "phone")
data class PhoneEntity(
    @Id
    val id: String,
    val events: EventEntity
) {
    companion object {
        operator fun invoke(phone: Phone) =
            with(phone) {
                PhoneEntity(
                    id,
                    EventEntity(events)
                )
            }
    }

    fun to(): Phone =
        Phone(
            id,
            events.to()
        )

    @Entity
    sealed class EventEntity {
        companion object {
            operator fun invoke(event: Event) =
                when (event) {
                    is ErrorEvent -> with(event) {
                        ErrorEventEntity(
                            name,
                            error,
                            time
                        )
                    }
                }
        }

        fun to(): Event =
            when (this) {
                is ErrorEventEntity ->
                    with(this) {
                        ErrorEvent(name, error, time)
                    }
            }

        data class ErrorEventEntity(
            val name: String,
            val error: Throwable,
            val time: Instant
        ) : EventEntity()
    }
}

We tried working around this by using the DatastoreCustomConversions thinking we could specify how to go about going from the saved Entity to the Event but we still end up with the same error. While debugging it looks like the Custom Converter is being registered in the DatastoreTemplate so unless theres something else we're missing, I think it's correct.

val EVENT_ENTITY_CONVERTER: Converter<PhoneEntity.EventEntity, Event> = object : Converter<PhoneEntity.EventEntity, Event> {
    override fun convert(eventEntity: PhoneEntity.EventEntity): Event {
        return eventEntity.to()
    }
}

@Configuration
class ConverterConfiguration {
    @Bean
    fun datastoreCustomConversions(): DatastoreCustomConversions {
        return DatastoreCustomConversions(
            listOf(
                EVENT_ENTITY_CONVERTER
            )
        )
    }
}

I've tried changing the Converter implementation a couple different times with no difference.

So primary questions:

1. Is there something we are doing obviously wrong with the Custom Converter that isn't allowing it to help with that process or is the converter unable to fix this issue?
2. If the converter won't help, is there something else we can do to maintain using a Sealed class on our Events that will allow us to be able to deserialize them from datastore?
3. Is there an alternative way we should be doing this that would be better or any obvious design flaws in our structure?

At present we have a somewhat work around but it involves changing our EventEntity into a data class and then taking all the possible fields for all the children events and making them nullable on the EventEntity. Then each construction ends up with a bunch of nulls for the unrelated fields when going back and forth (Event -> EventEntity and EventEntity -> Event). Which feels like a less flexible solution and just kind of ugly.

Any suggestions or help would be appreciated, thanks