Fixing Error: "InvalidArgumentError: Shapes of all inputs must match: values[0].shape = [16] != values[1].shape = [64]"

Question

I want to do single object detection on images in python. I created the following neural network with Keras Functional API:

IMG_SHAPE: Tuple = (1000, 750, 3)
BATCH_SIZE: int = 16

def build_model(img_shape: Tuple[int, int, int]) -> Model:  

    input_img = Input(shape=IMG_SHAPE)

    for k in [32,64,128]:
        x = BatchNormalization()(input_img)
        x = Conv2D(filters=k, kernel_size=3, padding="same")(x)
        x = Activation("relu")(x)
        x = MaxPool2D()(x)

    x = GlobalAveragePooling2D()(x)
    x = Flatten()(x)
    for k in [64,32]:
        x = LayerNormalization()(x)
        x = Dense(units=k)(x)
        x = Activation("relu")(x)

    output = Dense(units=4)(x)
    outputs = Activation("sigmoid")(output)

    model = Model(
        inputs=input_img,
        outputs=outputs,
        name="marker_detection_model"
    )
    model.save(MODELS_PATH)

    return model

Since I have limited amount of images I decided to use Keras ImageDataGenerator class to apply DataAugmentation as following:

# DATA AUGMENTATION
# create dataframe from image folder
df = pd.DataFrame([[str(f.absolute())] for f in libPath(IMG_DIR).glob("*.jpg")], columns=["filename"])
# add dummy column
df["coordinates"] = 0

# ImageDataGenerator class generates batches of tensor image data with real time data augmentation
train_datagen = ImageDataGenerator(
    rescale=1./255,
    channel_shift_range=100,
    brightness_range=(0.6, 0.6),
    validation_split=0.2
    )

train_generator = train_datagen.flow_from_dataframe(
    dataframe=df,
    directory=IMG_DIR,
    x_col="filename",
    y_col="coordinates",
    batch_size=BATCH_SIZE,
    class_mode="raw",
    target_size=IMG_SHAPE,
    subset="training"
)

validation_generator = train_datagen.flow_from_dataframe(
    dataframe=df,
    directory=IMG_DIR,
    x_col="filename",
    y_col="coordinates",
    batch_size=BATCH_SIZE,
    class_mode="raw",
    target_size=IMG_SHAPE,
    subset="validation"
)

After this DataAugmentation I start training my neural network where schedule_fn, schedule_fn2 and schedule_fn3 are simply customized functions that regulate learning rate after certain amount of epochs.

The shape of the images that serve as input data for the network is: (16, 1000, 750, 3) where 16 is the batch size, 1000 the image height, 750 the image width and 3 for RGB.

model = build_model(IMG_SHAPE)

schedules = [schedule_fn, schedule_fn2, schedule_fn3] 
for schedule in schedules:

    model_log_dir = os.path.join(MODEL_LOG_DIR, f"model{schedule.__name__}")

    model.compile(
        loss="mse",
        optimizer=Adam(lr=0.0005),
        metrics=[MeanIoU(num_classes=2)]
    )

    # callbacks
    early_stopping_callback = EarlyStopping(
        monitor="val_accuracy",
        patience=30,
        verbose=1,
        restore_best_weights=True,
        min_delta=0.0005
    )

    lrs_callback = LearningRateScheduler(
        schedule,
        verbose=1
    )

    tensorboard_callback = TensorBoard(
        log_dir=model_log_dir,
        histogram_freq=1,
        write_graph=True
    )

    history = model.fit(
        train_generator,
        steps_per_epoch=train_generator.samples // BATCH_SIZE,
        validation_data=validation_generator,
        validation_steps=validation_generator.samples // BATCH_SIZE,
        epochs=EPOCHS,
        callbacks=[lrs_callback, early_stopping_callback, tensorboard_callback],
        shuffle=True,
        verbose=1
    )

    history.history

    test_scores = model.evaluate_generator( # maybe change back to .evaluate()
        generator=validation_generator,
        steps=validation_generator.samples // BATCH_SIZE,
        verbose=1
    )

    print("Test Loss:", test_scores[0])
    print("Test Accuracy:", test_scores[1])

    predictions = model.predict(
        x=validation_generator,
        batch_size=BATCH_SIZE,
        callbacks=[early_stopping_callback, lrs_callback, tensorboard_callback],
        verbose=1
    )

Sadly the training starts and instantly throws me this Error:

Found 320 validated image filenames.
Found 80 validated image filenames.
2021-09-15 14:20:18.958934: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-09-15 14:20:18.959107: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-09-15 14:20:19.366981: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2021-09-15 14:20:20.067835: I tensorflow/core/profiler/lib/profiler_session.cc:136] Profiler session initializing.
2021-09-15 14:20:20.067854: I tensorflow/core/profiler/lib/profiler_session.cc:155] Profiler session started.
2021-09-15 14:20:20.067877: I tensorflow/core/profiler/lib/profiler_session.cc:172] Profiler session tear down.
======================================================================================
==============================FIT MODEL ON TRAINING DATA==============================
======================================================================================
2021-09-15 14:20:20.911492: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
Epoch 1/10

Epoch 00001: LearningRateScheduler reducing learning rate to 0.001.
Traceback (most recent call last):
  File "/Users/move37/VSCodeProjects/.../.../train.py", line 136, in <module>
    history = model.fit(
  File "/opt/anaconda3/envs/sciendisenv/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 1100, in fit
    tmp_logs = self.train_function(iterator)
  File "/opt/anaconda3/envs/sciendisenv/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 828, in __call__
    result = self._call(*args, **kwds)
  File "/opt/anaconda3/envs/sciendisenv/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 888, in _call
    return self._stateless_fn(*args, **kwds)
  File "/opt/anaconda3/envs/sciendisenv/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2942, in __call__
    return graph_function._call_flat(
  File "/opt/anaconda3/envs/sciendisenv/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1918, in _call_flat
    return self._build_call_outputs(self._inference_function.call(
  File "/opt/anaconda3/envs/sciendisenv/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 555, in call
    outputs = execute.execute(
  File "/opt/anaconda3/envs/sciendisenv/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError:  Shapes of all inputs must match: values[0].shape = [16] != values[1].shape = [64]
         [[node confusion_matrix/stack_1 (defined at Users/move37/VSCodeProjects/.../.../train.py:136) ]] [Op:__inference_train_function_3597]

Function call stack:
train_function

2021-09-15 14:20:30.501136: W tensorflow/core/kernels/data/generator_dataset_op.cc:107] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated.
         [[{{node PyFunc}}]]

I tried to apply solutions from other posts but nothing helped so far. Anyone who has some idea on what's the issue here? I'd be very grateful for any kind of help and in case I haven't posted this issue properly I want to apologize (this is my first attempt asking something on Stackoverflow)

NOTEWORTHY EDIT: WHEN I CHANGE THE BATCH_SIZE THE ERROR CHANGES IN A CERTAIN WAY. E.G.:

with batch_size = 16 tensorflow throws me the error: "InvalidArgumentError: Shapes of all inputs must match: values[0].shape = [16] != values[1].shape = [64]"

with batch_size = 8 tensorflow throws me the error: "InvalidArgumentError: Shapes of all inputs must match: values[0].shape = [8] != values[1].shape = [32]"

Any ideas of what this is about are very much appreciated!

Answer 1

SOLVED:

In model.compile(loss="mse", optimizer=Adam(lr=0.0005), metrics=[MeanIoU(num_classes=2)]) the metrics-parameter MeanIoU is initialized wrong. The error got caused by the wrong number of classes.

Answer 2

Not sure if that's the answer but in model.fit you haven't set batch_size which, if not set defaults to 32 as stated here