MultiResourceItemReader with a custom delegate keeps reading the same file

Question

Hello there java wizards,

I am having a lot of trouble getting into the spring batch. Straight to the point for now. I need to process all files (xmls) from a folder and them write them back, with a small addition. The problem was that I want to preserve the input filename. The solution that I have for this is a MultiResourceItemReader that delegates to a custom Itemreader that in turn calls the StaxEventItemReader and returns a custom item that holds the marshalled xml and the filename.

Question: only the same file is read in an infinite loop and another strange thing, there are 10 retries every time. I know that one solution would be that the reader to return null after the file was once read, but that means I need to keep a list of processed files ?
I guess I'll do that for now, but I really would like something smarter.

the job config:

<batch:job id="job1">
    <batch:step id="step1"  >           
        <batch:tasklet transaction-manager="transactionManager" start-limit="100" >
            <batch:chunk reader="reader" writer="writer" commit-interval="10" />
        </batch:tasklet>
     </batch:step>
</batch:job> 

<bean id="reader" class="org.springframework.batch.item.file.MultiResourceItemReader">
<property name="resources" value="files/*.xml" />
<property name="delegate" ref="myItemReader" />
</bean>

my item read method, basically:

public class MyItemReader implements ResourceAwareItemReaderItemStream<MyItem>, ApplicationContextAware {


public MyItem read() throws Exception, UnexpectedInputException,
        ParseException, NonTransientResourceException {

    StaxEventItemReader<JAXBElement<RootObject>> reader = new StaxEventItemReader<JAXBElement<RootObject>>();
    reader.setResource(currentResource);
    reader.setFragmentRootElementName("RootObject");

    // ... create jaxb unmarshaller

    reader.setUnmarshaller(unmarshaller);

    reader.setSaveState(true);
    reader.afterPropertiesSet();        

    reader.open(executionContext);

    JAXBElement<RootObject> jaxbElem = reader.read();

    MyItem item = new MyItem();

    item.setFilename(currentResource.getFile().getName());
    item.setJaxbElement(jaxbElem);

    return item;
}
}

Can anyone straighten me out here?

Solution So in the end I just keep a list of read files and return null if it's already read. As for the 10 reads at once, well, that is the chunk's size so it makes sense.

Answer 1

I don't think you want to be creating a new reader inside of your custom reader. I don't know if it's causing your problem but that doesn't seem right (and you're not closing it after reading).

You can use Spring to initialize your JAXB Context and then just inject it into your custom reader:

http://static.springsource.org/spring-ws/site/reference/html/oxm.html

Example:

<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:oxm="http://www.springframework.org/schema/oxm"
    xsi:schemaLocation="http://www.springframework.org/schema/oxm http://www.springframework.org/schema/oxm/spring-oxm.xsd
        http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">

    <bean id="myItemReader" class="com.example.MyItemReader">
        <property name="unmarshaller" ref="jaxbMarshaller"/>
    </bean>

    <oxm:jaxb2-marshaller id="jaxbMarshaller">
        <oxm:class-to-be-bound
            name="com.example.RootObject" />
    </oxm:jaxb2-marshaller>
</beans>

Then in your reader's read() method, just use the unmarshaller...

public MyItem read() throws Exception, UnexpectedInputException,
        ParseException, NonTransientResourceException {

    Source source = new StreamSource(resource);
    JAXBElement<RootObject> jaxbElem = unmarshaller.unmarshal(source);

    MyItem item = new MyItem();

    item.setFilename(resource.getFile().getName());
    item.setJaxbElement(jaxbElem);

    return item;
}

Edit

OK, I think the problem is with the read() method then. According to the Javadoc for ItemReader, when all items in the reader are exhausted, the read() method should return null... I don't think you're doing that so it is reading indefinitely.

I think finding a way to extend FlatFileItemReader or StaxEventItemReader is a better way to go... would something like this not work?

<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:batch="http://www.springframework.org/schema/batch"
    xmlns:oxm="http://www.springframework.org/schema/oxm"
    xsi:schemaLocation="http://www.springframework.org/schema/oxm http://www.springframework.org/schema/oxm/spring-oxm.xsd
        http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch.xsd
        http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">

    <batch:job id="job1">
        <batch:step id="step1"  >           
            <batch:tasklet transaction-manager="transactionManager" start-limit="100" >
                <batch:chunk reader="reader" writer="writer" commit-interval="10" />
            </batch:tasklet>
         </batch:step>
    </batch:job> 

    <bean id="reader" class="org.springframework.batch.item.file.MultiResourceItemReader">
        <property name="resources" value="files/*.xml" />
        <property name="delegate" ref="myItemReader" />
    </bean>

    <bean id="myItemReader" class="com.example.MyItemReader">
        <property name="unmarshaller" ref="jaxbMarshaller"/>
        <property name="fragmentRootElementName" ref="RootObject"/>
    </bean>

    <oxm:jaxb2-marshaller id="jaxbMarshaller">
        <oxm:class-to-be-bound
            name="com.example.RootObject" />
    </oxm:jaxb2-marshaller>
</beans>

Reader:

public class MyItemReader<T> extends StaxEventItemReader<T>{

    private Resource resource;

    @Override
    public void setResource(Resource resource) {
        this.resource = resource;
    }

    @Override
    protected T doRead() throws Exception {
        T jaxbElem = (T) super.doRead();

        MyItem item = new MyItem();

        item.setFilename(resource.getFile().getName());
        item.setJaxbElement(jaxbElem);

        return (T) item;
    }

}

Answer 2

It is necessary to let spring know when you have completely read the file. To do that you need to return null.

Keep a boolean flag to indicate if file processed already. If processed return null. and set the flag to false again.

boolean isRead=false;

MyItem read(){
 if(isRead){
  isRead=false;
  return null;
 }
 MyItem item=null;
 if(!isRead){
   isRead=true;
   //DO read.
    item=new Item();// Item to read....
  }
return item;
}