tensorflowjs how to get inner layer output in a cnn prediction

Question

I am looking at tensorflow.js CNN example from tfjs.

The testing repo can be found here: testing repo.

Is there any way I can get outputs from each layer?

 async showPredictions() {
    const testExamples = 1;
    // const testExamples = 100;
    const batch = this.data.nextTestBatch(testExamples);

    tf.tidy(() => {
        const output: any = this.model.predict(batch.xs.reshape([-1, 28, 28, 1]));

        output.print();
        const axis = 1;
        const labels = Array.from(batch.labels.argMax(axis).dataSync());
        const predictions = Array.from(output.argMax(axis).dataSync());

        // ui.showTestResults(batch, predictions, labels);
    });
}

Above is the prediction method from the tfjs example, but only the last layer is printed. How can I get outputs from each layer (including conv, max pooling and fully connect layers) in a prediction?

Answer 1

This Observable Notebook provides a detailed example of how to get the internal activations of a tf.Model instance in TensorFlow.js: https://beta.observablehq.com/@nsthorat/visualizing-activations-of-mobilenet-with-tensorflow-js

The basic idea behind it is constructing new tf.Model instances using the same input as the original model, but different outputs. Those outputs are the outputs of the individual layers of the original tf.Model instance. Something like

const newModel = tf.model({inputs: oldModel.inputs, outputs: oldModel.layers[n].output); 
const layerActivation = newModel.predict(inputValue);

Answer 2

thanks for the answer.

I have found another way to get the input and output data. I have modified the tfjs files in the node modules and attach the last input data and output data to each layer. Thus after each prediction, I can directly access input and output for each layer.

This approach works only before tfjs 0.11.6, the file to be changed is :

/node_modules/@tensorflow/tfjs-layers/dist-es6/engine/executor.js

Add this two line:

fetch.sourceLayer.outputData = output[0].dataSync(); 
fetch.sourceLayer.inputData = inputValues[0].dataSync();

Answer 3

I guess this is the solution!!

I was trying make the Model in convencional way but it was not possible to get two outputs, otherwise if you to use APPLY method it is possible to get two outputs like this: const [firstLayer, secondLayer] = model.predict(xs);

const input_nodes = 5;
const hidden_nodes = 8;
const output_nodes = 4;    
const input = tf.input({shape: [input_nodes]});
const denseLayer1 = tf.layers.dense({ units: hidden_nodes, activation: 'sigmoid' });
const denseLayer2 = tf.layers.dense({ units: output_nodes, activation: 'softmax' });
const output1 = denseLayer1.apply(input);
const output2 = denseLayer2.apply(output1);
const model = tf.model({inputs: input, outputs: [output1, output2]});

let inputs = [1, 2, 3, 4, 5];
const xs = tf.tensor2d([inputs]);
const [firstLayer, secondLayer] = model.predict(xs);

console.log(denseLayer1.getWeights().length)
denseLayer1.getWeights()[1].print()
console.log(denseLayer2.getWeights().length)

// PRINT

firstLayer.print();
secondLayer.print();

// OR console

console.log(firstLayer.flatten().arraySync());
console.log(secondLayer.flatten().arraySync());

Answer 4

You could create a feature extraction model that has all the layers in its output. This model will share layers with the source model.

// Create a model that has all the layers in its output.
const featExtractionModel = tf.model({
  inputs: model.input,
  outputs: model.layers.map(layer => layer.output)
});

// Execute model.
const extractedFeatures = featExtractionModel.predict(inputData);

// Now we have featExtractionModel.outputNames and extractedFeatures.

// Example: Get the output of an inner layer.
const innerLayerName = "inner_layer";
const innerLayerIndex = featExtractionModel.outputNames.indexOf(innerLayerName);
const innerLayerOutput = extractedFeatures[innerLayerIndex];

You will end up with an array of layer names and an array of layer outputs.