'Apache Beam: Integration test with unbounded PCollection with PAssert
I want test to that messages from a PubSub topic are parsed correctly into the protobuf structure.
The issue is that PubSubIO.Read is an unbounded source and the test does not terminate on its own.
One option which I tried is to terminate the pipeline manually by setting BlockOnRun=false and calling pipeline.cancel() but in this case the PAssert checks do not fire and any failing test pass.
What is the correct way to test elements of an unbounded PCollection with PAssert?
@Test
public void TestThatPublishedMessagesAreParsedCorrectly() throws IOException {
MyMessage testMessage = TestUtils.makeNewMessage();
String subscriptionName = initPubSubTopicWithMessages(testMessage);
Pipeline pipeline = createTestPipeline(getPubSubEmulatorRoot());
PCollection<MyMessage> messages = pipeline.apply(
PubsubIO
.readProtos(MyMessage.class)
.fromSubscription(subscriptionName));
PAssert.that(messages).containsInAnyOrder(testMessage);
PipelineResult result = pipeline.run();
result.waitUntilFinish(Duration.standardSeconds(5));
result.cancel();
}
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
