How to add a delay between DNN model layers at runtime?

Question

I am trying to analyze the impact of DNN layers (e.g. Convoluted, pooling, Fully-connected) on CPU performance by aggregating the values from performance counters. Since large DNNs (e.g. VGG16) is composed of many layers and its difficult to visualize the difference between consecutive layers, i wanted to add a "delay" between two consecutive layers during the DNN execution.

For example, the following is the first block of VGG16 implementation in keras:

# Block 1
x = layers.Conv2D(64, (3, 3), activation='relu', padding='same', name='block1_conv1')(
img_input)
x = layers.Conv2D(64, (3, 3), activation='relu', padding='same', name='block1_conv2')(x)
x = layers.MaxPooling2D((2, 2), strides=(2, 2), name='block1_pool')(x)

so I tried to add a delay using the lambda function as follows:

def sleep_layer(x):
time.sleep(5)
return x

# Block 1 with delay layer
x = layers.Conv2D(64, (3, 3), activation='relu', padding='same', name='block1_conv1')(
img_input)
x = tf.keras.layers.Lambda(sleep_layer, name="delay_layer1")(x)
x = layers.Conv2D(64, (3, 3), activation='relu', padding='same', name='block1_conv2')(x)
x = tf.keras.layers.Lambda(sleep_layer, name="delay_layer2")(x)
x = layers.MaxPooling2D((2, 2), strides=(2, 2), name='block1_pool')(x)
x = tf.keras.layers.Lambda(sleep_layer, name="delay_layer3")(x)

However, this doesn't translate to delay during the DNN runtime but rather when running the python script (the saved model doesn't contain any delays).

What is the right approach to add a delay between DNN layers at runtime?

Answer 1

I think it will be better to use time.sleep(5) directly instead of calling as layer. You can see the difference by commenting time.sleep(5) statements.

start = time.time()
# Block 1
x = layers.Conv2D(64, (3, 3), activation='relu', padding='same', name='block1_conv1')(
img_input)
time.sleep(5)
x = layers.Conv2D(64, (3, 3), activation='relu', padding='same', name='block1_conv2')(x)
time.sleep(5)
x = layers.MaxPooling2D((2, 2), strides=(2, 2), name='block1_pool')(x)
dur = time.time() - start
print(dur)