'How to change logic from foreach loop to batch in Mule 3
I am experimenting to convert a foreach logic that already works into batch job. In mule 4 it is quite simple, only in Mule 3 unfortunately not.
The idea of my experiment is that with the ids I got from a previous HTTP request call, to do new HTTP request call using the recieved id's quickly after each others. It could be 1000 ids (1000 requests) that need to be passed.
How could I do this? Is there a better way?
Below I added both logics;
Foreach version
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:http="http://www.mulesoft.org/schema/mule/http"
xmlns:dw="http://www.mulesoft.org/schema/mule/ee/dw"
xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
xmlns:spring="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/ee/dw http://www.mulesoft.org/schema/mule/ee/dw/current/dw.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd">
<http:request-config name="HTTP_Request_Configuration_Service"
protocol="HTTPS" host="${service.host}"
port="${service.port}"
basePath="?$filter=id eq #[payload]"
doc:name="HTTP Request Configuration" responseTimeout="${service.timeout}">
<http:basic-authentication username="${service.user}" password="${service.password}" preemptive="true"/>
</http:request-config>
<foreach doc:name="For Each">
<set-payload value="1234f15c-g4fb-511f-ba41-0ced971ec747,f6cb2052-3520-47c0-ba53-202029c8c5df,0ca68b3b-b2c0-4dba-a54a-2bbef4e8bcf2,12a8c48c-c221-4c10-82ab-c2222b5b2cac,3f394fda-d687-4d68-bb27-f532ca650803" doc:name="Incoming payload"/>
<dw:transform-message doc:name="Fetch the id">
<dw:set-variable variableName="id"><![CDATA[%dw 1.0
%output application/java
---
payload.value.id joinBy "," default ""]]></dw:set-variable>
</dw:transform-message>
<set-payload value="#[#[java.util.Arrays.asList(flowVars.id.split(','))]]" doc:name="Transform to array"/>
<foreach doc:name="For Each">
<http:request config-ref="HTTP_Request_Configuration_Service" path="/request/change/serviceRequest" method="GET" doc:name="HTTP Request Service">
<http:request-builder>
<http:query-params expression="{"?$filter": "id eq #[payload]"}"/>
</http:request-builder>
</http:request>
<dw:transform-message doc:name="Append payload">
<dw:set-variable variableName="outcome"><![CDATA[%dw 1.0
%output application/json
---
flowVars.outcome ++ payload]]></dw:set-variable>
</dw:transform-message>
</foreach>
</foreach>
</mule>
BATCH JOB version
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:file="http://www.mulesoft.org/schema/mule/file" xmlns:batch="http://www.mulesoft.org/schema/mule/batch" xmlns:http="http://www.mulesoft.org/schema/mule/http"
xmlns:dw="http://www.mulesoft.org/schema/mule/ee/dw"
xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
xmlns:spring="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/ee/dw http://www.mulesoft.org/schema/mule/ee/dw/current/dw.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/file http://www.mulesoft.org/schema/mule/file/current/mule-file.xsd
http://www.mulesoft.org/schema/mule/batch http://www.mulesoft.org/schema/mule/batch/current/mule-batch.xsd">
<http:request-config name="HTTP_Request_Configuration_Service"
protocol="HTTPS" host="${service.host}"
port="${service.port}"
basePath="?$filter=id eq #[payload]"
doc:name="HTTP Request Configuration" responseTimeout="${service.timeout}">
<http:basic-authentication username="${service.user}" password="${service.password}" preemptive="true"/>
</http:request-config>
<foreach doc:name="For Each">
<dw:transform-message doc:name="Transform Message">
<dw:set-payload><![CDATA[%dw 1.0
%output application/json
---
{
"@odata.context": "https://scrum-caller.com/repo/v1/inside#Changes",
"value": [
{
"id": "8db55441-6255-4d24-8d39-658536985214",
"number": "0w-30",
"Desc": "maintain"
},
{
"id": "11111111-6666-2222-3g3g-854712547412",
"number": "5w-40",
"Desc": "on prod"
},
{
"id": "1ab32c5b-ffs3-3243-74fv-3376218042bb",
"number": "5w-30",
"Desc": "on test"
}
]
}]]></dw:set-payload>
</dw:transform-message>
<dw:transform-message doc:name="Fetch the id">
<dw:set-variable variableName="id"><![CDATA[%dw 1.0
%output application/java
---
payload.value.id joinBy "," default ""]]></dw:set-variable>
</dw:transform-message>
<set-payload value="#[#[java.util.Arrays.asList(flowVars.id.split(','))]]" doc:name="Transform to array"/>
<set-session-variable variableName="outcome" value="#[[]]" doc:name="Outcome var array"/>
<batch:execute name="batch-job-flow" doc:name="Batch Execute"/>
</foreach>
<batch:job name="batch-job-flow">
<batch:process-records>
<batch:step name="Batch_Step">
<http:request config-ref="HTTP_Request_Configuration_Service" path="/request/change/serviceRequest" method="GET" doc:name="HTTP Request Service">
<http:request-builder>
<http:query-params expression="{"?$filter": "id eq #[payload]"}"/>
</http:request-builder>
</http:request>
<dw:transform-message doc:name="Append payload">
<dw:set-variable variableName="outcome"><![CDATA[%dw 1.0
%output application/json
---
flowVars.outcome ++ payload]]></dw:set-variable>
</dw:transform-message>
</batch:step>
<batch:step name="Batch_Step2">
<file:outbound-endpoint path="C:\Local\test" outputPattern="test.json" responseTimeout="10000" doc:name="File"/>
</batch:step>
</batch:process-records>
</batch:job>
</mule>
Solution 1:[1]
Your batch version is incorrect conceptually. For example you can not add values to a flow or session variable in a batch. Note that Batch works with records and outside flows.
Additionally you can not control in which thread each record is going to be processed. That means that the output may be overwritten or corrupted if you try to write it to a file. You need to use an output that is able to receive data concurrently, like a database. Then you should be able to extract and combine the output into a JSON file.
Note that foreach will also work with any number of records. You may have other problems like performance because it is not concurrent in Mule 3, but it can be in Mule 4.
Unrelated, I don't understand the strange transformations to try to get a list of ids. It seems that the simple DataWeave expression payload.value.*id should be enough to extract that from the input, but I may be missing something.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 |