time.sleep() not pausing timer in video

Question

Background

I have a video with people walking around different homes. I implemented 3 different timers (for 3 different people) that begin timing when the people are inside the bboxes of the homes and pause when they are outside of the bboxes of the homes. For reference, I have included a visual of what the first frame of the video looks like.

Problem

The check implemented in the code below works fine - it returns True when a person's bbox is inside of a home bbox and returns False otherwise. I'm having difficulties when it comes to pausing the time as a person is walking around outside of the home bbox (i.e. when a person's bbox is not inside of a home bbox). As you can see in the code below, I used time.sleep(1) to pause the timer for a second each time a False is returned, however, this doesn't seem to work. It just keeps the timer running in the background. For example, if person 1's timer is at 15 seconds before a False is returned, its timer should pause. Then, when a True is returned, its timer should resume its time from 15 seconds and increment. However, in my case right now, it keeps running the timer in the background so when a True is returned after a False, the timer suddenly shows up as 24 seconds instead of resuming from where it left off before the time.sleep(). How do I go about solving this issue?

# boxes_houses = [(#, #, #, #), (#, #, #, #), ...]

while cap.isOpened():

    # [...]

    def isInside(person, home):
        # Top-left corner
        if home['x1'] < person['x1'] and home['y1'] < person['y1']:
            # Bottom-right corner
            if person['x2'] + (person['x2'] - person['x1']) < home['x2'] + (home['x2'] - home['x1']) and person['y2'] + (person['y2'] - person['y1']) < home['y2'] + (home['y2'] - home['y1']):
                return True
        else:
            return False

    cnt_person = 0
    cnt_house = 0

    for box_person, box_house in zip(boxes_persons, boxes_houses):
        x_p1 = int(box_person[0])
        y_p1 = int(box_person[1])
        x_p2 = int(box_person[2])
        y_p2 = int(box_person[3])
        person_coords = {'x1': x_p1, 'y1': y_p1, 'x2': x_p2, 'y2': y_p2}
        cv2.rectangle(frame, (x_p1, y_p1), (x_p1 + x_p2, y_p1 + y_p2), (0, 0, 0), 2, 1)
        cv2.putText(frame, "House: {}".format(cnt_house), (x1, y1 - 10), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (0, 0, 0), 1)

        x_h1 = int(box_house[0])
        y_h1 = int(box_house[1])
        x_h2 = int(box_house[2])
        y_h2 = int(box_house[3])
        cv2.rectangle(frame, (x_h1 , y_h1), (x_h1 + x_h2, y_h1+ y_h2), (0, 0, 255), 2, 1)
        cv2.putText(frame, "Person: {}".format(cnt_person ), (int(box_house[0]), int(box_house[1] - 5)), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (0, 0, 255), 1)

        isinside_checks = []
        for house in houses:
            isinside_checks.append(isInside(person_coords, house))

        ### CHECK ###
        
        if any(inside_checks): #if persons inside the home bbox
            # print ("Person", cnt_person, ": True\n")

            if cnt_person==0:
                cv2.putText(main_frame, "Person 0 Time: {}".format(end_time-start_time), (450, 500), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (0,0,0),1)
            elif cnt_person==1:
                cv2.putText(main_frame, "Person 1 Time: {}".format(end_time-start_time), (450, 500), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (0,0,0),1)
            elif cnt_person==2:
                cv2.putText(main_frame, "Person 2 Time: {}".format(end_time-start_time), (450, 500), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (0,0,0),1)

        else: #if persons outside the home bbox
            print ("Person", cnt_person, ": False\n") 
            time.sleep(1) # should pause the time

        cnt_person = cnt_person + 1
        cnt_house = cnt_house + 1
                       
    # show frame
    cv2.imshow('MultiTracker', frame)

    if cv2.waitKey(1) & 0xFF == 27:  
        break