'What is the best way to debug the processElement part of the DoFn in dataflow

I am struggling with debugging my codes under processElement of DoFn in dataflow. I don't think I can step through this part, since it will start multiple threads and run even on my local machine (DirectRunner), any suggestion?

javagoogle-cloud-dataflowjavascriptnode.jstypescriptvue.jswebpack


Solution 1:[1]

So you did not give any information on your environment. First, I see differences between beam-sdk and Versions < 2. When using version 1.9.0 and running with DirectRunner and debug-mode in Eclipse, one can simply set a breakpoint inside the DoFn and you can debug as usual. With the beam-version, for now, I wasn't able to debug inside DoFn. Perhaps this helps you.

Solution 2:[2]

The way I do it is make sure only a single input is going to enter the pipeline thus running your code only once. For example if your pipeline reads from PubSub you can register to a DEV subscription when debugging and send a single message to debug.

Solution 3:[3]

Press F9 which is resume program in intellij and F8 in Eclipse/STS

Solution 4:[4]

You can try deleting your node_modules folder with rm -rf node_modules and the re-install everything with npm i, maybe something got messed up with npm.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Malte
Solution 2 srfrnk
Solution 3 Antakshari Salgaonkar
Solution 4 dennisbot

Related Questions