how to apply tranfer learning and retune a deep model on higher/lower quality but similar dataset?

Question

I am working on the classification of Medical Chest-Xrays images where I am using ResNet50v2. For now, I am doing classification. I have done the following experiments.

I have four different but same category datasets (normal/pneumonia)

1. Load pre-train ResNet using ImageNet weights and train the classification layers by adding Globalpool, Dropout layers, and change output as I required. (Get nice results)
2. Finetune by opening all layers to train model.trainable = True and retrain. (Get Nice results)
3. Again I finetune on a new dataset but the results are not that satisfactory.

I have below queries that I am trying to search out but failed.

• I am doing the right Transfer Learning?
• I am adding new layers at the classifier end during model loading as per my requirement is it okay?
• Should I open all layers to model.trainable = True every time when loading the saved model?
• Should I change hyperparameters during the training on the new dataset?

I have a pipeline like this: ---->Fresh New Training---->Finetunnig---->Finetune on new Dataset---->Finetune on new Dataset.

How I tackle this problem to improve my results.
I am sharing my model loading code:

Model Loading

    image_size = 512
    input_shape = (image_size, image_size, 3)
    
    epochs = 50
    batch_size = 16
    
    pre_trained_model = tf.keras.applications.ResNet50V2(input_shape=input_shape, include_top=False, weights="imagenet")
        
    for layer in pre_trained_model.layers:
        layer.trainable = False
        
    # last_layer = pre_trained_model.get_layer('block5_pool')
    # last_output = last_layer.output
        
    # Global Max pool due to finding 1x1 conv (heatmap) post task
    x = GlobalMaxPooling2D()(pre_trained_model.output)
    x = Dropout(0.3)(x)
    
    x = Dense(512, activation='relu')(x)
    # Add a final sigmoid layer for classification
    x = layers.Dense(2, activation='softmax')(x)
    
    model = Model(pre_trained_model.input, x)
    
    model.compile(loss='categorical_crossentropy',
                  optimizer=adam,
                  metrics=metric)
    
    model.summary()
history = model.fit(train_generator, validation_data=validation_generator,
                    steps_per_epoch=train_steps,                    
                    validation_steps=val_steps,
                    epochs=50, use_multiprocessing=True, callbacks=call_backs)

Finetunning model Loading

    dependencies = {
    'f1_m': f1_m,
    'recall_m': recall_m,
    'precision_m': precision_m,
    'specificity': specificity,
    
}
model = keras.models.load_model('/models/classifier/saved-model-50-0.90.hdf5', custom_objects=dependencies)
model.summary()

model.trainable = True
# Let's take a look to see how many layers are in the base model
print("Number of layers in the base model: ", len(model.layers))
len(model.trainable_variables)

for i, layer in enumerate(model.layers):
    print(i, layer.name, layer.trainable)

history = model.fit(train_generator, validation_data=validation_generator,
                    steps_per_epoch=train_steps,                    
                    validation_steps=val_steps,
                    epochs=50, use_multiprocessing=True, callbacks=call_backs)

Note:

I am adding all the required steps for improving results, e.g.

• augmentation
• dropout
• different call_backs (lr_decay, early_stop, checkpoint)
• play with learning_rate
• optimizers
• class-weights (classes are balance perfectly)
• multi-scale features pooling
• shuffle
• use BatchNormalization Thank your patience.

Answer 1

To answer your questions (in the same order):

• Yes, you are doing transfer learning.
• Yes, it is okay to add your own layers to the model as per your requirement.
• The recommended approach is to only make the newly added layers trainable and train for a few epochs. Then, make all the layers trainable and train again. If you add your own layers to the model and start training all the layers, large gradient updates to your newly added layers can mess up the pre-trained weights in the rest of the model as gradient updates are propagated backwards during backpropagation.
• Yes, ideally you should.

To improve your results, you may consider using other model architectures, or a different approach for solving the problem altogether.