Swagger exception when trying to resolve protcol buffer generated Java classes

Question

I am trying to generate a json rest api. My return Object to the client are classes generated from protocol buffers. So I tweacked jackson to convert the protobuf messages into json,so the rest api works. But when adding swagger, I get an exception when trying to create the swagger definition file. Here is how I try to create the definition file:

            SwaggerConfiguration oasConfig = new SwaggerConfiguration().openAPI(oas);

            Reader reader = new Reader(oasConfig);
            OpenAPI openAPI = reader.read(app.getClasses());
            swagger = Json.pretty(openAPI);

The Exception I am getting is

java.lang.IllegalArgumentException: Conflicting setter definitions for property "author": bisq.social.protobuf.PrivateTradeChatMessage$Builder#setAuthor(bisq.social.protobuf.ChatUser) vs bisq.social.protobuf.PrivateTradeChatMessage$Builder#setAuthor(bisq.social.protobuf.ChatUser$Builder) at com.fasterxml.jackson.databind.introspect.POJOPropertyBuilder._selectSetterFromMultiple(POJOPropertyBuilder.java:561)

Since the Java classes being analyzed here are generated, I am not able to change them.

Is there a way to make swagger work with protobuf generated classes?

This is using swagger 2.2.0 and jackson 2.13.2