'Way to change value based on condition with previous validated data?

I cannot manage to implement in an efficient way a method that could change values in dataframes based on difference with previous "validated" data.

I have a dataframe with dates as index and assets as columns, full of weights. I would like to implement a method that would update the weight only if the difference with the previous "validated" weight is equal or higher than a certain threshold.

For example, with 2 assets, with a threshold of 5, i would like to keep the weights flagged as OK :

A B
10 OK (first entry) 10 OK (first entry)
8 8
15 OK (previous data was 10 so diff=5) 17 OK (previous data was 10 so diff=7)
20 OK (previous data was 15 so diff=5) 20
23 23 OK (previous data was 17 so diff=6)

Meaning the final output i'd like is:

A B
10 10
10 10
15 17
20 17
20 23

I have tried the following, but it's only taking the difference with the previous data, so not exactly what I need.

pct_change_filter = np.where(acted_weight.ffill().fillna(0).diff().abs()>0.05, 1, np.nan)
filtered_final = acted_weight.multiply(pct_change_filter)

For information I have a few hundreds of columns and approximately 10K rows.

Many thanks !



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source