Inputing a torch 3d tensor into a keras.Sequential model

Question

Here I have a pytorch tensor object which I need to use for training a neural network. Can pytorch tensors be used for training a keras neural network and if so, what will be the value of the input_shape parameter when the training data has 3 dimensions?

Here's the code:

image, label = val_dset.__getitem__(20)
print(image, label)

Output:

tensor([[[0.8549, 0.8549, 0.8588,  ..., 0.8667, 0.8627, 0.8627],
         [0.8549, 0.8549, 0.8588,  ..., 0.8667, 0.8667, 0.8667],
         [0.8549, 0.8549, 0.8549,  ..., 0.8667, 0.8667, 0.8667],
         ...,
         [0.1216, 0.1412, 0.1686,  ..., 0.5490, 0.7529, 0.5686],
         [0.1451, 0.1843, 0.2667,  ..., 0.4510, 0.6627, 0.5765],
         [0.3098, 0.4078, 0.5098,  ..., 0.3529, 0.4588, 0.4431]],

        [[0.8549, 0.8549, 0.8588,  ..., 0.8902, 0.8824, 0.8824],
         [0.8549, 0.8549, 0.8588,  ..., 0.8863, 0.8824, 0.8824],
         [0.8549, 0.8549, 0.8588,  ..., 0.8863, 0.8824, 0.8784],
         ...,
         [0.1294, 0.1451, 0.1725,  ..., 0.4745, 0.6824, 0.4902],
         [0.1451, 0.1843, 0.2627,  ..., 0.3804, 0.5804, 0.5098],
         [0.2863, 0.3686, 0.4745,  ..., 0.3020, 0.3843, 0.3922]],

        [[0.8510, 0.8510, 0.8510,  ..., 0.8667, 0.8627, 0.8627],
         [0.8510, 0.8510, 0.8510,  ..., 0.8667, 0.8667, 0.8667],
         [0.8510, 0.8510, 0.8510,  ..., 0.8667, 0.8667, 0.8627],
         ...,
         [0.1294, 0.1451, 0.1765,  ..., 0.3804, 0.5294, 0.3882],
         [0.1490, 0.1882, 0.2627,  ..., 0.3137, 0.4588, 0.4039],
         [0.2902, 0.3647, 0.4588,  ..., 0.2471, 0.3176, 0.3059]]]) 1

image.shape

Output:

torch.Size([3, 224, 224])

For the network:

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers, callbacks, Input

model = keras.Sequential([
    Input(shape=(_)),
    layers.BatchNormalization(),
    layers.Dropout(0.5),
    layers.Dense(512, activation='relu'),
    layers.Dropout(0.5),
    layers.Dense(1024, activation='relu'),
    layers.Dropout(0.5),
    layers.Dense(1024, activation='relu'),
    layers.Dropout(0.5),
    layers.Dense(1024, activation='relu'),
    layers.BatchNormalization(),
    layers.Dropout(0.5),
    layers.Dense(26, activation='softmax')    
])

model.compile(
    loss='categorical_crossentropy',
    optimizer='adam',
    metrics=['accuracy']
)

early_stopping = keras.callbacks.EarlyStopping(
    patience=50,
    min_delta=0.005,
    restore_best_weights=True
)

Answer 1

You can get .numpy() then convert to tensor like below:

>>> torch_image = torch.rand(3, 224, 224)
>>> torch_image.shape
torch.Size([3, 224, 224])

>>> tf_image = tf.convert_to_tensor(torch_image.numpy())
>>> tf_image
<tf.Tensor: shape=(3, 224, 224), dtype=float32, numpy=
array([[[0.7935423 , 0.7479191 , 0.976204  , ..., 0.31874692,
         0.70065683, 0.77162033],
        ...,
        [0.19848353, 0.41827488, 0.5245047 , ..., 0.28861862,
         0.5350955 , 0.6847391 ],
        [0.57963634, 0.8628217 , 0.0179103 , ..., 0.19654012,
         0.38167596, 0.5232694 ]]], dtype=float32)>

Or If you have multiple images then want to convert to tensor and train your model, you can use tf.data.Dataset.from_tensor_slices this function by default does this for you and convert torch to tensor like below:

(In the below example I create 100 random arrays as images)

import torch
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers, callbacks, Input

torch_image = torch.rand(100, 3, 224, 224)
torch_label = torch.randint(1,5,(100,))

dataset = tf.data.Dataset.from_tensor_slices((torch_image, torch_label))
dataset = dataset.repeat(10).batch(5)

# for check type Tensor or Torch
# tf_image , label_image = next(iter(dataset))
# >>> tf_image
# <tf.Tensor: shape=(5, 3, 224, 224), dtype=float32, numpy=...>

model = keras.Sequential([
    Input(shape=(3,224,224,)),
    layers.Flatten(),
    layers.Dense(64, activation='relu'),
    layers.BatchNormalization(),
    layers.Dropout(0.5),
    layers.Dense(1, activation='softmax')    
])

model.compile(
    loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

early_stopping = keras.callbacks.EarlyStopping(
    patience=50, min_delta=0.005, restore_best_weights=True)

model.fit(dataset, epochs=10)

Output:

Epoch 1/10
200/200 [==============================] - 7s 37ms/step - loss: 0.0000e+00 - accuracy: 0.2500