Does pre-trained VGG-16 work with LeakyRelu?

Question

Does pre-trained VGG-16 with ImageNet weights work with LeakyRelu activation function?

Our input has negative values hence we would like to use LeakyRelu.

Sample code below.

VGG-16 using ReLu:

# CREATE VGG16
vgg = VGG16(include_top=False, pooling = 'max', weights='imagenet', input_shape=(96,96,3)) 
vgg.get_config()

# freeze weights in vgg layers 1 to 10
for layer in vgg.layers[:-10]: 
    layer.trainable = False

# Check if any layer is not frozen
for each in vgg.layers:
    print(each, each.trainable)

# Put a \newline to make it neat    
print('')

# Look at the summary of VGG16
vgg.summary()

VGG-16 using LeakyRelu:

# CREATE VGG16
vgg = VGG16(include_top=False, pooling = 'max', weights='imagenet', input_shape=(96,96,3)) 

for layer in vgg.layers[:-10]:
    if (hasattr(layer,'activation'))==True:
        layer.activation = LeakyReLU(alpha=leaky_relu_alpha)

# freeze weights in vgg layers 1 to 10
for layer in vgg.layers[:-10]: 
    layer.trainable = False

# Check if any layer is not frozen
for each in vgg.layers:
    print(each, each.trainable)

# Put a \newline to make it neat    
print('')

# Look at the summary of VGG16
vgg.summary()

Answer 1

If the VGG is pertained with relu, then leakyrelu may not work. Because the network layer after the relu activation function never saw feature map with negative values and therefore may lead to low performance. However, the impact of leakyrelu on performance highly depends on your task. I suggest you fine-tune the pre-trained VGG with leakyrelu activation and use a small learning rate for several epochs. This should address the problem of performance degradation.

Another way of handling your negative input is normalizing it into value within range of [0,1] or some other ranges.