How to profile a keras predict call with tensorboard

Question

I would like to have a timing profile/trace of a predict call to get an estimate of how fast my model can perform inference.

Right now I'm using:

log_dir="logs/profile/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")

tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, profile_batch = 1)

x_test, y_test = next(iter(training_ds))
_ = unet.predict(x_test, verbose=1, callbacks=[tensorboard_callback])

But the profiling tab does not show up in tensorboard. What am I missing here?

Answer 1

First, see if CUPTI is correctly loading. In a terminal you should see something like:

2019-12-13 12:01:47.617853: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcupti.so.10.0

If it didn't find the CUPTI libraries, make sure that your LD_LIBRARY_PATH is set correctly. $ echo $LD_LIBRARY_PATH should return something like:

/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64

If this is all set, run the following snippet of code, assuming you have described your model in tensorflow/keras:

# Set up logging.
stamp = datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
logdir = 'logs/trace/%s' % stamp
writer = tf.summary.create_file_writer(logdir)

tf.summary.trace_on(graph=True, profiler=True)
# Forward pass
input, label = next(iter(dataset)) # tf DataSet object
your_model(input)
with writer.as_default():
    tf.summary.trace_export(name="model_trace", step=0, profiler_outdir=logdir)

Now, the next final step is critical for viewing the trace in Tensorboard: you have to view Tensorboard in Chrome for it to parse the .trace file correctly.

Answer 2

If you are using virtual environments, make sure you don't mix things up. See my answer to another question.

Also, there are 4 methods to write profiling data. As you tried callbacks before, you should give the other methods a try: Overview of profiling methods.

Answer 3

First of all, make sure that CUPTI is correctly loading. Recently I have tried to find the solution and I used following:

import tensorflow as tf
logs = "../logs/"
for input_data in datas:
    with tf.profiler.experimental.Profile(logs):
        out_pred = model.predict(input_data)
        pass