'Custom reactive ETL process in spring-data-rest

I have an task, where I would like to provide some preprocessing of external data and pass them to frontend. Actually simple architecture image can be worth of many words:

Deployment diagram with interactions

I am getting all jobs from some project and converting them to SimpleJob (job id and status).

Particular pieces of puzzle:

  • getting data from gitlab: use of gitlab4j-api (actually custom HttpClient calls would do the thing
  • using some custom @Service classes to have @Timed annotation work (reporting to prometheus) and some methods annotated using @Cacheable to have request to gitlab cached (cached are only "final" results, like finished pipelines
  • transforming Job onto SimpleJob is done using java stream/map/collect/...
  • generating changes in collection of transformed SimpleJobs is done using ObservableMap from javafx-base dependency.
  • emitting changes using SSE (server side events) is done using one Sinks.many().multicast().onBackpressureBuffer() and publishing it on some endpont like this:
    @RequestMapping(path = "/stream-flux-read", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
    public Flux<ServerSentEvent> sse() {return sink.asFlux().map(e -> ServerSentEvent.builder(e).build());}

Question: Is there any nice (spring) way of managing the observable part? Now it is just a bunch of code where I add custom listeners to changes of Map<JobId, SimpleJob>, which does pass events to sink (which can be observed using sse endpoint). I also add things manually to given map after they are processed. Is there anything like "ObservableCache", which can be used in such way, that I would not manually write (maintain!) code for adding things to given map and converting listening events onto events in sink/flow?



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source