'Custom reactive ETL process in spring-data-rest
I have an task, where I would like to provide some preprocessing of external data and pass them to frontend. Actually simple architecture image can be worth of many words:
I am getting all jobs from some project and converting them to SimpleJob (job id and status).
Particular pieces of puzzle:
- getting data from gitlab: use of gitlab4j-api (actually custom HttpClient calls would do the thing
- using some custom @Service classes to have @Timed annotation work (reporting to prometheus) and some methods annotated using @Cacheable to have request to gitlab cached (cached are only "final" results, like finished pipelines
- transforming Job onto SimpleJob is done using java stream/map/collect/...
- generating changes in collection of transformed SimpleJobs is done using ObservableMap from javafx-base dependency.
- emitting changes using SSE (server side events) is done using one
Sinks.many().multicast().onBackpressureBuffer()and publishing it on some endpont like this:
@RequestMapping(path = "/stream-flux-read", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<ServerSentEvent> sse() {return sink.asFlux().map(e -> ServerSentEvent.builder(e).build());}
Question: Is there any nice (spring) way of managing the observable part? Now it is just a bunch of code where I add custom listeners to changes of Map<JobId, SimpleJob>, which does pass events to sink (which can be observed using sse endpoint). I also add things manually to given map after they are processed. Is there anything like "ObservableCache", which can be used in such way, that I would not manually write (maintain!) code for adding things to given map and converting listening events onto events in sink/flow?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
