What to do if neural network always performs poorly even after addressing overfitting?

Question

I have a medical image dataset of ~10K 256x256 images with which I am training a deep neural classifier for disease classification. I have been working with popular CNNs like InceptionV3 and ResNets.

These models have achieved validation set accuracies in the 50-60% range and I noticed that they were overfitting. So to improve the performance, I then tried common strategies like a dropout in the dense layers, smaller learning rates, and L2 regularization. After these modifications showed no reduction in overfitting, I next moved to smaller and simpler architectures with just 2-3 convolution layers + 1 FC classification layer which I thought would mitigate the issue. However, with the simpler models, the learning curves still showed signs of overfitting. Particularly, when training for 100 epochs, the models would have similar train and validation losses for the first 20-30 epochs, but then diverge after that.

I'm not sure what other strategies I can experiment with at this point and I'm worried that trying more experiments aimlessly is inefficient. Should I just accept that the models cannot generalize to this task well?

Additionally, FYI, the dataset is imbalanced, but I have dealt with this using data augmentation and a weighted cross-entropy loss as well but no real difference.

Answer 1

1. Try to use modern classification approaches like transformers or efficientnets - their accuracy is higher. To compare different modern architectures please use paperswithcode.

2. Augmentations, regularizations are must-have in training process, doesn't matter if balanced or imbalanced data you have.

3. You can try to make over- or undersampling of your data to get better results

4. Try to use warmup and learning rate schedules, this improves the convergence of the model

Answer 2

Understand what PREROLLING in GStreamer means. (TL;DR, the pipeline will wait for each sink to have at least one buffer ready before it can switch to PLAYING)

Also keep in mind that data buffers are passed around via reference, not copies. Now think about what happens with lookahead in the encoder (TL;DR, it will consume frames without outputting enything, until enough lookahead frames have entered the encoder. The diplay brnach howver will not consume anything)

If the display sink does not consume anything it will block upstream, because where would the data go elsewhere. If upstream is blocked, it cannot deliver anymore data to the encoder which it requires to unblock itself. Hence everything stalls.