'How to use both destination_prefix_naming and default_file_naming in Apache Beam WriteToFiles?
I have an Apache Beam pipeline that is writing to files via:
with beam.Pipeline(options=options) as p:
my_pipeline = (
p
| "ReadTable" >> beam.io.Read(source)
| "WriteSomeFiles" >> beam.io.fileio.WriteToFiles(
path="/path/prefix/to/directory/",
destination=lambda record: record["filename"],
file_naming=beam.io.fileio.destination_prefix_naming(),
sink=JsonSink())
)
This successfully names the files with the "filename" entry in my record. How can I also set the suffix of the file? E.g. with default_file_naming like:
default_file_naming(prefix="my-prefix", suffix=".json")
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
