'Value in checkpoint could not be found in the restored object
I'm hitting an error when trying to restore a tensorflow model after the training has completed. In particular, I have a GAN and I separately save generator and discriminator weights to be restored later on. I do like this:
Save model weights for generator and discriminator CNNs (both are classes inheriting from tf.keras.Model):
g_ckp_dir = os.path.join(checkpoint_dir, 'generator')
d_ckp_dir = os.path.join(checkpoint_dir, 'discriminator')
generator.save_weights(os.path.join(g_ckp_dir, 'checkpoint'))
discriminator.save_weights(os.path.join(f"{d_ckp_dir}", 'checkpoint'))
Then, I load as:
g_ckp_dir = os.path.join(checkpoint_dir, 'generator')
d_ckp_dir = os.path.join(checkpoint_dir, 'discriminator')
generator.load_weights(os.path.join(g_ckp_dir, 'checkpoint'))
discriminator.load_weights(os.path.join(f"{d_ckp_dir}", 'checkpoint'))
However, the model does not seem to be restored correctly because predictions are not meaningful, and I get also the warnings below:
WARNING:tensorflow:Detecting that an object or model or tf.train.Checkpoint is being deleted with unrestored values. See the following logs for the specific values in question. To silence these warnings, use
status.expect_partial(). See https://www.tensorflow.org/api_docs/python/tf/train/Checkpoint#restorefor details about the status object returned by the restore function.WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).to_image_ops.4.kernel WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).to_image_ops.4.bias WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).to_image_ops.5.kernel WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).to_image_ops.5.bias WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).to_image_ops.6.kernel WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).to_image_ops.6.bias WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).spade_blocks.0.conv1.kernel WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).spade_blocks.0.conv1.bias WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).spade_blocks.0.conv2.kernel WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).spade_blocks.0.conv2.bias WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).spade_blocks.1.conv1.kernel ...
What is wrong with this? Why do I get such an error?
I use Python 3.8.10, and TensorFlow 2.8.0
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
