OSError: SavedModel file does not exist when attempting to load downloaded TF2 model

Question

using detect_fn = tf.keras.models.load_model(PATH_TO_SAVED_MODEL) I get the error above. My filepath is generated when downloading the model:

def download_model(model_name, model_date):
    base_url = 'http://download.tensorflow.org/models/object_detection/tf2/'
    model_file = model_name + '.tar.gz'
    model_dir = tf.keras.utils.get_file(fname=model_name,
                                        origin=base_url + model_date + '/' + model_file,
                                        untar=True)
    return str(model_dir)

and is ~/.keras/datasets/centernet_hg104_1024x1024_coco17/saved_model with a saved_model.pb file within the directory. It seems to be asking for a saveed_model.pb file but does not accept the directory I provide which contains the file. Any idea what the issue is? is it an error with the tf.keras.utils.get_file() downloading or is it an error with my loading?