Read one record/item and write multiple records/items using spring batch

Question

I did some searching but couldn't find any sample/example.

I've a requirement where geo coordinates from one table (input) are read, processed to generate POI's associated to coordinate. So one geo coordinate will result in one or more POI's that needs to be inserted into another table (output).

I'm currently using a JdbcCursorItemReader and JdbcBatchItemWriter to read one item/record and write one item/record. There is also an ItemProcessor that generates the POI's for a give geo coordinate.

Does a custom JdbcBatchItemWriter help me achieve this?

Any ideas? TIA.

Answer 1

What you are really looking for is called a Splitter pattern:

Here is how it is defined in Spring Integration:

A Splitter is a type of Message Endpoint whose responsibility is to accept a Message from its input channel, split that Message into multiple Messages, and then send each of those to its output channel. This is typically used for dividing a "composite" payload object into a group of Messages containing the sub-divided payloads.

Configuration is extremely simple:

<channel id="inputChannel"/>

<splitter id="splitter" 
  ref="splitterBean" 
  method="split" 
  input-channel="inputChannel" 
  output-channel="outputChannel" />

<channel id="outputChannel"/>

<beans:bean id="splitterBean" class="sample.PojoSplitter"/>

Or you can use annotations:

@Splitter
List<LineItem> extractItems(Order order) {
    return order.getItems()
}

You can of course write your own JdbcBatchItemWriter if it feels simpler. However Spring Integration already does it for you.

You can use Spring Integration JDBC Support => jdbc:inbound-channel-adapter / jdbc:outbound-channel-adapter and the above splitter to achieve what you want and.. simplicity.

Answer 2

I did this by extending the Writer class (in my case HibernateItemWriter). I see one answer describes how to use a 'splitter'. If anyone has a working example of how that might work in an environment using the spring-boot-starter-parent, I would love to see it. For what I am doing (creating a List from a single record) it would be so much easier if Spring provided a write method that handled a List of Lists.

Here is how I extended the Writer to handle multiple writes for every line read/processed: Spring-Batch Multi-line record Item Writer with variable number of lines per record

Answer 3

I had a similar case and i could solve it with the following reusable code:

@Slf4j
public class ListItemWriter<T> implements ItemWriter<List<T>> {

  private ItemWriter<T> delegate;

  @Setter private int chunkSize = 0;

  public ListItemWriter(ItemWriter<T> delegate) {
    this.delegate = delegate;
  }

  @Override
  public void write(List<? extends List<T>> items) throws Exception {
    if (chunkSize == 0) {
      chunkSize = items.size();
    }
    List<T> allItems = items.stream().flatMap(Collection::stream).collect(Collectors.toList());
    log.info("writing {} items", allItems.size());
    List<List<T>> partitions = ListUtils.partition(allItems, chunkSize);
    for (List<T> partition : partitions) {
      delegate.write(partition);
    }
  }
}

Reader: A
Processor: A -> List<B>
Writer: new ListItemWriter(ItemWriterForB)