How to disable e.g. Dropout in tf.keras.Model to generate activation maximation images using transfer learning

Question

I am using transfer learning and keras.applications.InceptionV3. I manage to train the model successfully.

However, when I want to generate "activation maximisation" images (e.g. the input image that maximizes the activation of one of the custom classes, ref eg https://arxiv.org/pdf/1512.02017v3.pdf ) I struggle to use the pre-trained model since I do manage to use it in "fit" mode and disable all dropouts etc.

What I do is that I combine the pre-trained model in a tf.keras.Sequential to do gradient descent on the weights of the first layer (the input image).

Despite setting base_model.trainable = False however it seems as if the pre-trained model is put into training mode (although weights are not updated) when using model.fit(data) on the outer sequential model.

Is there any way to force the base_model (a child of a Sequential) to be in "predict" mode when calling fit on the outer?

Answer 1

I just came across the same question. After reading some documentation and having a look on the source code of TensorFlows implementations of tf.keras.layers.Layer, tf.keras.layers.Dense, and tf.keras.layers.BatchNormalization I got the following understanding.

If training = False is passed on calling the layer, it will run in inference mode. This has nothing to do with the attribute trainable, which means something different. It would probably lead to less misunderstanding, if they would have called it training_mode instead.

When doing Transfer Learning or Fine Tuning training = False should be passed on calling the base model itself. As far as I saw until now this will only affect layers like tf.keras.layers.Dropout and tf.keras.layers.BatchNormalization and will have not effect on the other layers.

Running in inference mode via training = False will result in tf.layers.Dropout not to apply the dropout rate at all.

As tf.layers.Dropout has no trainable weights, setting the attribute trainable = False will have no effect at all,