How to map features from two different data using regressor for classification?

Question

I am trying to build a Keras model to implement to approach explained in this paper.

Context of my implementation:

I have two different kinds of data representing the same set of classes(labels) that needs to be classified. The 1st kind is Image data, and the second kind is EEG data (a time series sequence).

I know that to classify image data we can use CNN models like this:

model.add(Conv2D(filters=256, kernel_size=(11,11), strides=(1,1), padding='valid'))
model.add(Activation('relu'))
model.add(Dense(1000))
model.add(Activation('relu'))

model.add(Dropout(0.4))
# Batch Normalisation
model.add(BatchNormalization())

# Output Layer
model.add(Dense(40))
model.add(Activation('softmax'))

And to classify sequence data we can use LSTM models like this:

model.add(LSTM(units = 50, return_sequences = True))
model.add(Dropout(0.2))

model.add(Flatten())
model.add(Dense(32, activation='relu'))
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(40, activation='softmax'))

But the approach of the paper above shows that EEG feature vectors can be mapped with image vectors through regression like this:

1. The first approach is to train a CNN to map images to corresponding EEG feature vectors. Typically, the first layers of CNN attempt to learn the general (global) features of the images, which are common between many tasks, thus we initialize the weights of these layers using pre-trained models, and then learn the weights of the last layers from scratch in an end-to-end setting. In particular, we used the pre-trained AlexNet CNN, and modified it by replacing the softmax classification layer with a regression layer (containing as many neurons as the dimensionality of the EEG feature vectors), using Euclidean loss as the objective function.
2. The second approach consists of extracting image features using pre-trained CNN models and then employ regression methods to map image features to EEG feature vectors. We used our fine-tuned AlexNet as feature extractors by reading the output of the last fully connected layer, and then applied several regression methods (namely, k-NN regression, ridge regression, random forest regression) to obtain the predicted feature vectors

I am not able to comprehend how to code the above two approaches. I have never used a regressor for feature mapping and then do classification. Any leads on this are much appreciated.

Answer 1

In my understanding the training data consists of (eeg_signal,image,class_label) triplets.

1. Train the LSTM model with input=eeg_signal, output=class_label. Loss is crossentropy.
2. Peel off the last layer of the LSTM model. Let's say the pre-last layer's output is a vector of size 20. Let's call it eeg_representation.
3. Run this truncated model on all your eeg_signal inputs, save the output of eeg_representation. You will get a tensor of [batch, 20]
4. Take that AlexNet mentioned in the paper (or any other image classifier), peel off the last layer. Let's say the pre-last layer's output is a vector of size 30. Let's call it image_representation.
5. Stich a linear layer to the end of the previous layer. This layer will convert image_representation to eeg_representation. It has 20 x 30 weight.
6. Train the stiched model on (image, eeg_representation) pairs. Loss is the Euclidean distance.
7. And now the fun part: Stich together model trained in step 7. and the peeled off part of model trained in step 1. If you input an image, you will get class predictions. This sound like not a big deal (because we do image classification all the time), but if this is really working, it means that this is a "prediction that is running through our brains" :) Thank you bringing up this question and linking the paper.

I feel I just repeated what's in your question and in the the paper. I would be beneficial to have some toy dataset to be able to provide code examples.

Here's a Tensorflow tutorial on how to "peel off" the last layer of a pretrained image classification model.