'Normalization of data using MinMaxScaler over a list of dataframes

I been trying to normalize my data using MinMaxScaler method. However, I have a list of data frames (20 columns) and every one of them include different number of data readings. For example, the first one has 23 cell. Then, if we get into each cell, we will see 1500x24 data (Not every cell has the same number of rows, it's changeable). My plan is to find the min and max across the entire data by scanning those 20 categories (col variable). If I use scikitlearn MinMaxScaler, it only does the normalization considering that specific sub-category, not over the whole data (Or I could not achieve it). Probably, I am also having some issues in importing data because I could not convert it into exactly the array form I wanted. The plan is to convert them into array, normalize the data and then transform them into grayscale RGB images.

Thus, the problem is to find the max and min across all 20 list of dataframes and then normalize every sub-category based on that max and min value. Could anyone give some ideas?

enter image description here enter image description here

col = ['UN_DTE','UN_DTF','UN_SM05','UN_SM10','UN_SM15','D1_DTE', 'D1_DTF', 'D1_SM05', 'D1_SM10', 'D1_SM15', 'D2_DTE','D2_DTF','D2_SM05','D2_SM10','D2_SM15','D3_DTE','D3_DTF','D3_SM05','D3_SM10','D3_SM15']
df = []
for i in range(len(col)):
    df.append(pd.DataFrame(mat[col[i]]))
# Normalize the data
scaler = MinMaxScaler()
scaler.fit(df[0][0][0])
scaled_features = scaler.transform(df[0][0][0])
pd.DataFrame(scaled_features).describe()



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source