multiple image input training using dataset object

Question

How to use a dataset object as input in the model.fit() training loop, for a model with multiple inputs? Trying to pass the dataset itself gives me the following error:

Failed to find data adapter that can handle input: (<class 'list'> containing values of types {"<class 'tensorflow.python.data.ops.dataset_ops.MapDataset'>"}), <class 'NoneType'>

My case here:

I have a multiple input model built with keras The inputs are named 'First', 'Second' and 'Third'

I have an image dataset in keras-style:

main_directory/
...class_a/
......a_image_1.jpg
......a_image_2.jpg
...class_b/
......b_image_1.jpg
......b_image_2.jpg

I create the dataset object using tf.keras.utils.image_dataset_from_directory:

train_dataset = image_dataset_from_directory(train_dir,
                                             shuffle=False,
                                             label_mode='categorical',
                                             batch_size=hyperparameters["BATCH_SIZE"],
                                             image_size=IMG_SIZE)

Now, each image is divided in 3 parts, each part serving as input to each of the inputs of the model. I take care of that using some map functions. This is not relevant tot he problem and I will not include it. I cannot use the cropping layers included in TF because of unrelated reasons.

I then try to start the training loop:

 history = model.fit([train_dataset1, 
                      train_dataset2, 
                      train_dataset3,
                      ],
                      epochs=epochs, 
                      callbacks=callbacks,
                      validation_data=validation_dataset
                      validation_steps=steps
                      )

And here is where I get the error. I have tried some other approaches, like using a dict instead of a list. The problem seems to be that when training a model with multiple inputs, the fit() loop expects data to come as a list for x-values and a list for y-values, but I haven't been able to split the dataset object into the required formats

I have read many topics on this, but all use datasets that are created using the tf.data.Dataset.from_tensor_slices() method, which is not applicable in my case

Additionally, there is no indication of how the validation dataset has to be structured (at least according to the model.fit() documentation)

I have found some guidance saying that the validation dataset must have the same number of input/outputs as the training datasets (makes sense), but again, no indication on how to build or feed the validation dataset for a multiple input model

Answer 1

As I stated in a comment above, I managed to solve this issue, but there seems to be a bug in the way the Keras fit() training loops handles the input from a zipped dataset, so if you need to do this, you'll have to wait until it's fixed or write your own training loop.

How to approach this

I ended up solving this issue in the following way:

I created 3 separate dataset objects. Then to feed it to the model.fit() training loop, I used the tf.data.Dataset.zip() (as Almog David pointed out in the comments) method to create a single dataset containing the 3 separate datasets:

train_dataset = tf.keras.preprocessing.image_dataset_from_directory(train_dir,
                                             shuffle=False,
                                             label_mode='categorical',
                                             batch_size=32,
                                             image_size=IMG_SIZE)
validation_dataset = tf.keras.preprocessing.image_dataset_from_directory(validation_dir,
                                             shuffle=False,
                                             label_mode='categorical',
                                             batch_size=32,
                                             image_size=IMG_SIZE)

def resize_data1(images, classes):
    return (tfimgcrop(images,
                        offset_height=0,
                        offset_width=0,
                        target_height=64,
                        target_width=64),
                    classes)
def resize_data2(images, classes):
    return (tfimgcrop(images,
                        offset_height=0,
                        offset_width=64,
                        target_height=64,
                        target_width=64),
                    classes)
def resize_data3(images, classes):
    return (tfimgcrop(images,
                        offset_height=0,
                    offset_width=128,
                    target_height=64,
                    target_width=64),
                classes)

train_dataset_unb = train_dataset.unbatch()
train_dataset1 = train_dataset_unb.map(resize_data1)
train_dataset2 = train_dataset_unb.map(resize_data2)
train_dataset3 = train_dataset_unb.map(resize_data3)
train_dataset_zip = tf.data.Dataset.zip((train_dataset1, train_dataset2, train_dataset3))

validation_dataset_unb = validation_dataset.unbatch()
validation_dataset1 = validation_dataset_unb.map(resize_data1)
validation_dataset2 = validation_dataset_unb.map(resize_data2)
validation_dataset3 = validation_dataset_unb.map(resize_data3)
validation_dataset_zip = tf.data.Dataset.zip((validation_dataset1, validation_dataset2, validation_dataset3))

Validating the approach by testing what the zipped datasets return in each call:

Printing elements in a for() loop using the tf.data.Dataset.as_numpy_iterator() method:

for element in train_dataset_zip.as_numpy_iterator():
print("element", element)

Outputs:

element [[[x1, y1, z1], [c1,]], [[x2, y2, z2], [c2,]], [[x3, y3, z3], [c3,]]] 
element [[[x1, y1, z1], [c1,]], [[x2, y2, z2], [c2,]], [[x3, y3, z3], [c3,]]] 
[...]
element [[[x1, y1, z1], [c1,]], [[x2, y2, z2], [c2,]], [[x3, y3, z3], [c3,]]] 

Printing elements in a for() loop using the python intrinsic enumerate() function:

for idx, (ds1, ds2, ds3) in enumerate(train_dataset_zip):
    print("ds1: ", ds1)
    print("ds2: ", ds2)
    print("ds3: ", ds3)

Outputs:

ds1:  (<tf.Tensor: shape=(64, 64, 3), dtype=float32, numpy=[(large array with raw pixel values)], , dtype=float32)>, <tf.Tensor: shape=(1,), dtype=float32, numpy=array([0.], dtype=float32)>
ds2:  (<tf.Tensor: shape=(64, 64, 3), dtype=float32, numpy=[(large array with raw pixel values)], , dtype=float32)>, <tf.Tensor: shape=(1,), dtype=float32, numpy=array([0.], dtype=float32)>
ds3:  (<tf.Tensor: shape=(64, 64, 3), dtype=float32, numpy=[(large array with raw pixel values)], , dtype=float32)>, <tf.Tensor: shape=(1,), dtype=float32, numpy=array([0.], dtype=float32)>
ds1:  (<tf.Tensor: shape=(64, 64, 3), dtype=float32, numpy=[(large array with raw pixel values)], , dtype=float32)>, <tf.Tensor: shape=(1,), dtype=float32, numpy=array([0.], dtype=float32)>
ds2:  (<tf.Tensor: shape=(64, 64, 3), dtype=float32, numpy=[(large array with raw pixel values)], , dtype=float32)>, <tf.Tensor: shape=(1,), dtype=float32, numpy=array([0.], dtype=float32)>
ds3:  (<tf.Tensor: shape=(64, 64, 3), dtype=float32, numpy=[(large array with raw pixel values)], , dtype=float32)>, <tf.Tensor: shape=(1,), dtype=float32, numpy=array([0.], dtype=float32)>
[...]
ds1:  (<tf.Tensor: shape=(64, 64, 3), dtype=float32, numpy=[(large array with raw pixel values)], , dtype=float32)>, <tf.Tensor: shape=(1,), dtype=float32, numpy=array([0.], dtype=float32)>
ds2:  (<tf.Tensor: shape=(64, 64, 3), dtype=float32, numpy=[(large array with raw pixel values)], , dtype=float32)>, <tf.Tensor: shape=(1,), dtype=float32, numpy=array([0.], dtype=float32)>
ds3:  (<tf.Tensor: shape=(64, 64, 3), dtype=float32, numpy=[(large array with raw pixel values)], , dtype=float32)>, <tf.Tensor: shape=(1,), dtype=float32, numpy=array([0.], dtype=float32)>