How Can I Increase My CNN Model's Accuracy

Question

I built a cnn model that classifies facial moods as happy , sad, energetic and neutral faces. I used Vgg16 pre-trained model and freezed all layers. After 50 epoch of training my model's test accuracy is 0.65 validatation loss is about 0.8 .

My train data folder has 16000(4x4000) , validation data folder has 2000(4x500) and Test data folder has 4000(4x1000) rgb images.

1)What is your suggestion to increase the model accuracy?

2)I have tried to do some prediction with my model , predicted class is always same. What can cause the problem?

What I Have Tried So Far ?

1. Add dropout layer (0.5)
2. Add Dense (256, relu) before last layer
3. Shuff the train and validation datas.
4. Decrease the learning rate to 1e-5

But I could not the increase validation and test accuracy.

My Codes

train_src = "/content/drive/MyDrive/Affectnet/train_class/"
val_src = "/content/drive/MyDrive/Affectnet/val_class/"
test_src="/content/drive/MyDrive/Affectnet/test_classs/"

train_datagen = tensorflow.keras.preprocessing.image.ImageDataGenerator(
      rescale=1./255, 
      shear_range=0.2,
      zoom_range=0.2,
      horizontal_flip=True,
    
      )

train_generator = train_datagen.flow_from_directory(
        train_src,
        target_size=(224,224 ),
        batch_size=32,
        class_mode='categorical',
        shuffle=True
        )

validation_datagen = tensorflow.keras.preprocessing.image.ImageDataGenerator(
        rescale=1./255
        )

validation_generator = validation_datagen.flow_from_directory(
        val_src,
        target_size=(224, 224),
        batch_size=32,
        class_mode='categorical',
        shuffle=True
        )
conv_base = tensorflow.keras.applications.VGG16(weights='imagenet',
                  include_top=False,
                  input_shape=(224, 224, 3)
                  )
for layer in conv_base.layers:
  layer.trainable = False

model = tensorflow.keras.models.Sequential()

# VGG16 is added as convolutional layer.
model.add(conv_base)

# Layers are converted from matrices to a vector.
model.add(tensorflow.keras.layers.Flatten())

# Our neural layer is added.
model.add(tensorflow.keras.layers.Dropout(0.5))
model.add(tensorflow.keras.layers.Dense(256, activation='relu'))

model.add(tensorflow.keras.layers.Dense(4, activation='softmax'))

model.compile(loss='categorical_crossentropy',
              optimizer=tensorflow.keras.optimizers.Adam(lr=1e-5),
              metrics=['acc'])
history = model.fit_generator(
      train_generator,
      epochs=50,
      steps_per_epoch=100,
      validation_data=validation_generator,
      validation_steps=5,
      workers=8
      )

Loss and accuracy