RecyclerView adding extra space between items

Question

I've a grid of items that change their position every few seconds based on the input from the server. Upon every change, I update the adapter and the items move around in the grid.

I want to add space on top & bottom of an item if they're on second line or later lines. The items in first line of the grid have space only on the bottom of the item.

The problem I'm running into is upon every data change in the grid that results in item moving around Eg. Item that's on the second line is now the first item on the first line, the decoration space keeps increasing between items.

I'm removing previous decoration before adding new decoration but still the space between items keeps increasing.

UsersGrid.kt

var paddingDecorator: PaddingDecorator 
init {

   recyclerview = findViewById(R.id.list)
   paddingDecorator = PaddingDecorator(3)
   //setup binder and layout manager
   recyclerview.addItemDecoration(paddingDecorator)
}

fun bindData(newData: Data) {
     //update data in the adapter then
   
     recyclerView.removeItemDecoration(paddingDecorator)
     paddingDecorator = PaddingDecorator(3)
     
     recyclerview.addItemDecoration(paddingDecorator)   
}

private class PaddingDecorator(val columns: Int) {

     override fun getItemOffset(outRect: Rect, view:View, parent: RecyclerView, state: RecyclerView.State) {
        val itemPosition = parent.getChildAdapterPosition(view)

        if(itemPosition < columns) {
           outRect.top = 0
           outRect.bottom = EXTRA_MARGIN
        }
        else {
           outRect.top = EXTRA_MARGIN
           outRect.bottom = EXTRA_MARGIN
        }
     }

}

What am I doing wrong here? How can I prevent extra space being added upon every adapter data update?