Bad result when using VGG preprocess_input in the model

Question

I didn't put VGG16 preprocess_input in my ImageDataGenerator. But instead I put it in the model. The result of putting preprocess_input in the model is way worse than putting preprocess_input in ImageDataGenerator.

Putting preprocess input in the model

train_datagen = keras.preprocessing.image.ImageDataGenerator(
    # Some augmentation
    # without preprocessing_function=preprocess_input
)

val_datagen = keras.preprocessing.image.ImageDataGenerator(
    # without preprocessing_function=preprocess_input
)

train_generator = train_datagen.flow_from_directory(
    ...
)

val_generator = val_datagen.flow_from_directory(
    ...
)

keras.backend.clear_session()

base_model = VGG16(include_top=False, weights="imagenet")
for layer in base_model.layers:
  layer.trainable = False

classifier_model = keras.models.Sequential([
  keras.layers.Flatten(),
  keras.layers.Dense(128, activation="relu"),
  keras.layers.Dropout(0.2),
  keras.layers.Dense(64, activation="relu"),
  keras.layers.Dropout(0.2),                                         
])

input_layer = keras.layers.Input(shape=(128, 128, 3))
x = tf.cast(input_layer, tf.float32)
x = keras.layers.Lambda(keras.applications.vgg16.preprocess_input)(x) # Layer preprocess_input
x = base_model(x)
x = classifier_model(x)
output_layer = keras.layers.Dense(5, activation="softmax")(x)

model = keras.models.Model(inputs=[input_layer], outputs=[output_layer])

This model giving me accuracy and validation accuracy around 25%

Putting preprocess input in ImageDataGenerator

train_datagen = keras.preprocessing.image.ImageDataGenerator(
    # Some augmentation and rescale
    preprocessing_function=preprocess_input #add preprocessing_function=preprocess_input
)

val_datagen = keras.preprocessing.image.ImageDataGenerator(
    # rescale
    preprocessing_function=preprocess_input #add preprocessing_function=preprocess_input
)

train_generator = train_datagen.flow_from_directory(
    ...
)

val_generator = val_datagen.flow_from_directory(
    ...
)

base_model = VGG16(include_top=False, weights="imagenet", input_shape=[128, 128, 3])
for layer in base_model.layers:
  layer.trainable = False

classifier_model = keras.models.Sequential([
  keras.layers.Flatten(),
  keras.layers.Dense(128, activation="relu"),
  keras.layers.Dropout(0.2),
  keras.layers.Dense(64, activation="relu"),
  keras.layers.Dropout(0.2),                                         
])

model = keras.models.Sequential([
  base_model,
  classifier_model,
  keras.layers.Dense(5, activation="softmax")
])

This model giving me accuracy and validation accuracy around 85%

Did I do wrong in putting the preprocessing layer or something?