java.lang.IllegalStateException: LayoutNode should be attached to an owner when scrolling paginated Lazy column in Jetpack compose

Question

I am using LazyColumn with Paging 3 to load contacts, when I scroll the list its laggy and sometimes it throws the following exception

java.lang.IllegalStateException: LayoutNode should be attached to an owner
        at androidx.compose.ui.node.LayoutNodeKt.requireOwner(LayoutNode.kt:1407)
        at androidx.compose.ui.node.ModifierLocalConsumerNode.notifyConsumerOfChanges(ModifierLocalConsumerNode.kt:42)

Following is the code for LazyColumn:

val contactsFlow= viewModel.fetchContacts(context)
val lazyContacts = contactsFlow.collectAsLazyPagingItems()
    LazyColumn(
        modifier = Modifier.padding(top = 8.dp),
        verticalArrangement = Arrangement.spacedBy(8.dp)) {
        items(lazyContacts) { contact ->
            if(contact != null)
            {
                Contact(contact = contact, onContactClick = {
                    viewModel.addMemberFrom(it) { member ->
                        onContactClick(member)
                    }
                })
            }
        }
    }

Here is the viewModel code:

fun fetchContacts(context: Context): Flow<PagingData<Contact>> {
        return Pager(
            PagingConfig(10, enablePlaceholders = true, initialLoadSize = 20)
        ) {
            ContactsPagingSource(contactManager, context)
        }.flow.cachedIn(viewModelScope)

    }