'GeneratorDatasetOp:Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations
I have created a tf record and used it to train my model. During the training process, Whenever I am calling my data to perform the evaluation I am always getting the message
Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations.
During the creation of tf record, I try to include below code to optimize the dataset but this doesn't worked out
AUTO = tf.data.experimental.AUTOTUNE # used in tf.data.Dataset API
option_no_order = tf.data.Options()
option_no_order.experimental_deterministic = False
option_no_order.experimental_optimization.noop_elimination = True
option_no_order.experimental_optimization.apply_default_optimizations = True
Please suggest the steps that I can try out or follow to optimize the dataset
Solution 1:[1]
I got the same warning message when my model was created using distributed strategy. Creating the model with non-distributed strategy removed the warning. If you need to use distributed strategy, I think you need to use distributed dataset as well (https://www.tensorflow.org/api_docs/python/tf/distribute/DistributedDataset).
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Tuukka Salmi |