How to have side effect on object when deserializing with Jackson?

Question

Under Spring Boot 2.4.5 and using FasterXML's Jackson 2.11.4, I'm intending to obtain a collection of objects from a database via a MyBatis mapper that goes like this:

<select id="getMyCollection" resultType="MyObject">
    SELECT other_properties, DATE_COL AS "auditInfo.date", TIME_COL AS "auditInfo.time"
    FROM A_TABLE
</select>

The MyObject class has an auditInfo field like so:

public class MyObject{
    ...

    private AuditInfo auditInfo;
    
    public AuditInfo getAuditInfo() {
        return auditInfo;
    }

    public void setAuditInfo(AuditInfo auditInfo) {
        this.auditInfo = auditInfo;
    }

    ...
}

And AuditInfo has the following:

public class AuditInfo {
    private LocalDateTime dateTime = getCurrentLocalDateTime();
    private String user;

    public AuditInfo(Authentication auth) {
        this.user = getCurrentUserName(auth);
    }

    public AuditInfo() {
    }

    @JsonSerialize(using = MyLocalDateSerializer.class)
    public LocalDate getDate() {
        return this.dateTime != null ? this.dateTime.toLocalDate() : null;
    }

    public void setDate(Object date) {
        LocalDate d = null;

        if (date instanceof LocalDate) {
            d = (LocalDate) date;
        } else if (date instanceof Timestamp) {
            //This comes from the mapper
            d = LocalDate.from(((Timestamp) date).toLocalDateTime());
        }

        //SIDE EFFECT
        this.dateTime = combineIntoLocalDateTime(d, this.getTime());
    }

    @JsonSerialize(using = MyLocalTimeSerializer.class)
    public LocalTime getTime() {
        return this.dateTime != null ? this.dateTime.toLocalTime() : null;
    }

    public void setTime(Object time) {
        LocalTime t = null;

        if (time instanceof LocalTime) {
            t = (LocalTime) time;
        } else if (time instanceof Timestamp) {
            //This comes from the mapper
            t = LocalTime.from(((Timestamp) time).toLocalDateTime());
        }

        //SIDE EFFECT
        this.dateTime = combineIntoLocalDateTime(this.getDate(), t);
    }

    public Timestamp getDateTime() {
        return this.getTime() != null ? Timestamp.valueOf(dateTime) : null;
    }

    public void setDateTime(LocalDateTime dateTime) {
        this.dateTime = dateTime;
    }

    public String getUser() {
        return user;
    }

    public void setUser(String user) {
        this.user = user;
    }
}

Having these serializers:

public class LocalDateSerializer extends JsonSerializer<LocalDate> {
    @Override
    public void serialize(LocalDate tmp, JsonGenerator jsonGenerator, SerializerProvider serializerProvider) throws IOException {
        jsonGenerator.writeObject(Utils.TimeUtils.dateToEpochMillis(tmp));
    }
}

public static class LocalTimeSerializer extends JsonSerializer<LocalTime> {
    @Override
    public void serialize(LocalTime tmp, JsonGenerator jsonGenerator, SerializerProvider serializerProvider) throws IOException {
        jsonGenerator.writeObject(tmp.format(DateTimeFormatter.ofPattern("HHmmss")));
    }
}

I'm facing that the dateTime property of the AuditInfo method never gets updated when Jackson uses the date or time setters for deserializing, resulting in it always having the initial value, i.e., the current date and time. I've noted that after console-printing my object collection and the tmp of the LocalTimeSerializer's serializing method.

How could I have this side effect inside the setters and expect Jackson to keep that internal property updated?