'Iterate over multi-index pandas dataframe and add first index to variable if certain conditions are met

This is going to be a complex question about something I have struggled with for months:

The following is a much simplified pandas dataframe than I actually have, where 7, 9 and 1 being the column names and R1, R4, R2, R6, R9 and R7 being the first index and Tsi: 1, 2 and 3, being the second index:

df =         Tsi     7    9   
        R1   1     0.2  0.1  
        R4   1     0.4  0.6  
        R2   2     0.7  0.3  
        R6   2     0.3  0.1  
        R9   3     0.4  0.1  
        R7   3     0.4  0.6 

I need to iterate over each column and add the first indexes to a list or a dictionary (any sort of variable I can use to keep the indexes in), but only if they belong to the same Tsi-group (1, 2 or 3 here).

I also need to get the variances for the values of this Tsi-group and this way see if adding one neighboring value will lower the variance, for this column.

If the variance gets lower, the first index of the neighbor needs to be added to the variable. This means that R2 will be added to the variable if it lowers the variance calculated for R1 and R4 (in Tsi-group 1).

For Tsi-group 2, both neighbors will have to be tested to see if they decrease the variance: R4 (belonging to Tsi 1) and/or R9 (Tsi 3) will be added if they decrease the variance of R2 and R6.

And again for Tsi-group 3, R6 will be added if it decreases the variance.

Im picturing the variable to be a dictionary with the column names being the keys and the first indexes to be the values, as lists of lists. But any variable that could store column names and the indexes would be fine.

What I'm expecting for the first column would be the following dictionary:

TsiDict = {7 : [['R1','R4'], ['R2','R6','R4', 'R9'], ['R9','R7']], 9: [[R1,R4,R2,], ['R2','R6','R9'], ['R9','R7','R1']] }

As I said, I have spent so much time trying to figure out how to do this, but what I have tried thus far ruins one of the goals I am needing to accomplish with this, like ruining the ordering or not storing correct column names or index.

Any help would be immensely appreciated!



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source