'Autoencoder Custom Error Metric not working as intended

I have created a custom error metric to measure the success of an autoencoder implemented in Keras and sklearn. At a high level, it computes the loss for each of the output neurons separately (num_features = num_input_neurons = num_output_neurons), and returns the highest and lowest loss of the features over the whole testing dataset, as well as the names of the two features themselves.

The implementation of the custom loss functions look like this:

import keras.backend as K

def get_argmax_max(features, diff_list):
    return [features[K.argmax(diff_list)], K.max(diff_list).numpy()]

def get_argmin_min(features, diff_list):
    return [features[K.argmin(diff_list)], K.min(diff_list).numpy()]
# features = [list of feature names]
decoded_prediction = test_autoencoder.predict(testing_data)
individual_diff_list = list(abs(x-y) for x, y in zip(decoded_prediction, testing_data))
error_list = np.mean(individual_diff_list, axis=0)
max_err = get_argmax_max(features, error_list)
min_err = get_argmin_min(features, error_list)

I have added two columns to train and test - one completely random, and one uniform (eg: all 1s) - in order to test the success of this custom metric. My hypothesis is that the custom metric will identify the uniform column as having the minimum error (as it is very simple to learn) and the random column as having the maximum error (as it is not possible to learn well). In practice, however, this is not the case, and the metric often selects random other columns instead of the expected uniform and random columns. I have ensured that the range of the random column exceeds that of all columns in the dataset, tested with sufficiently large encoding dimension, and have also tried running for a large number of epochs to no avail.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source