Listing and deleting permissions using Google Drive API sometimes does not work

Question

I'm using the google drive API to retrieve the permissions on a folder and then delete them. Sometimes the delete works, but other times I get an error: <HttpError 404 when requesting https://www.googleapis.com/drive/v3/files/1gJKxxxxxxxxxxxLgpz/permissions/writeremail%40gmail.com?supportsAllDrives=true returned "Permission not found: [email protected].". Details: "[{'domain': 'global', 'reason': 'notFound', 'message': 'Permission not found: [email protected].', 'locationType': 'parameter', 'location': 'permissionId'}]">

        drive_service = build('drive', 'v3', credentials=creds)
        page_token = None
        while True:
            perms=drive_service.permissions().list(fileId = TEST_PERMISSIONS_FOLDER_ID, fields='permissions/id,permissions/displayName,permissions/role,permissions/emailAddress').execute()
            page_token = perms.get('nextPageToken', None)
            if page_token is None:
                break

        perm_id_list =[]
        for p in perms.get('permissions', []):
            if p.get('role') != 'owner': # put ids and emails for non-owner in a list
                perm_id_list.append({p.get('id'),p.get('emailAddress')})

        for id, email in perm_id_list: # do the delete using the id
            drive_service.permissions().delete(fileId = TEST_PERMISSIONS_FOLDER_ID, permissionId=id, supportsAllDrives = True).execute()

When I go into Drive and look at the share permissions, all attempting to be deleted are there. It's like the ids are not correct. Help! I'm so frustrated!