Is there anything wrong with the my Inception-V3? val_accuracy is not increasing and val_loss is not decreasing

Question

Can anyone help me to reduce val_loss and increase val_accuracy? Below is the code of my Inception-V3 model. I added two dense layers with 1024 nodes with ReLU activation function, 1 GlobalAverage2D and 1 Dropout layer. And 1 last fully-connected layer with softmax activation for classification purpose.

To increase val_accuracy and reduce val_loss. Do i need to remove any one of the following layers?

pretrained_model = tf.keras.applications.InceptionV3(
    input_shape=(224, 224, 3),
    include_top=False,
    weights='imagenet'
)

pretrained_model.trainable = False
inputs = pretrained_model.input
x = GlobalAveragePooling2D()(pretrained_model.output)
x = tf.keras.layers.Dense(1024, activation='relu')(x)
x = tf.keras.layers.Dense(1024, activation='relu')(x)
x = Dropout(0.5)(x)
outputs = tf.keras.layers.Dense(5, activation='softmax')(x)

model = tf.keras.Model(inputs=inputs, outputs=outputs)

model.compile(
    optimizer=Adam(learning_rate=0.0001),
    loss='categorical_crossentropy',
    metrics=['accuracy','AUC']
)

history = model.fit(
    train_images,
    validation_data=val_images,
    batch_size = 64,
    epochs=20,
    callbacks=[
        tf.keras.callbacks.EarlyStopping(
            monitor='val_loss',
            patience=5,
            restore_best_weights=True
        )
    ]
)

Below graphs shows the accuracy and loss obtained from the trained inception-V3 model. accuracy

loss