Runtime Error: tensorflow.python.framework.errors_impl.NotFoundError: Could not find metadata file. [[{{node LoadDataset/_1}}]] [Op:DatasetFromGraph]

Question

As in the tutorial, trying to execute tff.learning.build_federated_averaging_process(model_fn, client_optimizer_fn=lambda: tf.keras.optimizers.SGD(0.02)) but on orchestrator (server) with data saved on edge node (client) using tf.data.experimental.load() method:

@tff.tf_computation
def make_data():
    element_spec = collections.OrderedDict([('x', tf.TensorSpec(shape=(None, 784), dtype=tf.float32, name=None)),
             ('y', tf.TensorSpec(shape=(None,), dtype=tf.int32, name=None))])
    data = tf.data.experimental.load('./train_data', element_spec = element_spec)
    return data

However, I'm getting the following error:

W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at dataset_ops.cc:175 : Not found: Could not find metadata file.
         [[{{node LoadDataset/_1}}]]

TF data was saved using tf.data.experimental.save(train_data[0], './train_data') method. The implementation works when executed locally: tff.backends.native.set_local_execution_context()

python - 3.7

libraries versions:

tensorflow - 2.5.2

tensorflow-estimator - 2.5.0

tensorflow-federated - 0.19.0

Any help would be most appreciated.

Answer 1

When you decorate a Python function with @tff.tf_computation it will serialize the contents as a tf.Graph to be reused later. Frankly, I do not know how I/O like the experimental tf.data load logic interacts with it.

The recommended pattern would be to avoid that, and instead load the data in Python at the level where you are creating TFF computations, and pass the loaded dataset as an input to your tff.tf_computation or tff.federated_computation, with matching type signature (tff.types.SequenceType).