'Unable to pass java library path with beam multi-language pipelines
We are using a beam multi-language pipeline using python and java(ref https://beam.apache.org/documentation/sdks/python-multi-language-pipelines/). We are creating a cross-language pipeline using java. We have some external jar files that required a java library path. Code gets compiled properly and is able to create a jar file. When I run the jar file it creates a Grpc server but when I use the python pipeline to call External transform it is not picking up the java library path it picks the default java library path.
Tried -Djava.library.path=<path_to_dll> while running jar file. Tried System.setProperty(“java.library.path”, “/path/to/library”). (Ref https://examples.javacodegeeks.com/java-library-path-what-is-java-library-and-how-to-use/)
Tried JvmInitializer of beam to overwrite system property. (Ref https://examples.javacodegeeks.com/java-library-path-what-is-java-library-and-how-to-use/)
Tried to pull code beam open source and tried to overwrite system proprty before expansion starts. It overwrite but it is not picking correct java path when calls using python external transform. (ref https://github.com/apache/beam/blob/master/sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/ExpansionService.java)
Solution 1:[1]
You may want to use Artifacts staging.
By default, Portable Runner uses a pipeline class path to find the artifacts for staging to Job Server. It’s also possible to stage only required artifacts with a pipeline option: --filesToStage=<comma separated jars list>
It takes more time to stage artifacts when a pipeline runs for the first time. For the next attempts of pipeline run, it will check if the artifacts for staging has not been changed (using hashing) since last time and it will use already staged(cached) artifacts if no other artifacts staging is required.
Also, it’s possible to avoid an artifact staging at all if all required jars and other files will be available in the class path on workers in case of using Process SDK Harness or in the class path in Docker image in case of using Docker SDK Harness. In this case we need to leave --filesToStage option empty.
Solution 2:[2]
It sounds like you're trying to stage native code, and not just Jar files. You could try using custom containers instead. You would have to make sure the expansion service that you start up is configured to use these containers (e.g. via pipeline options).
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Alexey Romanenko |
| Solution 2 | robertwb |