'Weight Clustering in Pytorch
I have been trying to do weight clustering in PyTorch for my project. For this I have used the following very rudimentary code :
from sklearn.cluster import KMeans
# from kmeans_pytorch import kmeans, kmeans_predict
'''Going to go through all the layers --> obtain their weights--> use scikit learn to do the clustering--> replace these weight by their centroids'''
def weight_clustering(model):
model.to('cpu')
with torch.no_grad():
for name, params in model.named_parameters():
param_shape=list(params.size())
# print("the shape is ",param_shape)
weights=torch.flatten(params)
weights=params.reshape(-1,1)
kmeans = KMeans(n_clusters=3, random_state=0).fit(weights)
cluster_centers=torch.from_numpy(kmeans.cluster_centers_)
for i in range(0,len(kmeans.labels_)):
if kmeans.labels_[i]==0:
weights[i]=cluster_centers[0]
elif kmeans.labels_[i]==1:
weights[i]=cluster_centers[1]
elif kmeans.labels_[i]==2:
weights[i]=cluster_centers[2]
elif kmeans.labels_[i]==3:
weights[i]=cluster_centers[3]
else :
weights[i]=cluster_centers[4]
reshape_size_tuple=tuple(param_shape)
weights=weights.reshape(reshape_size_tuple)
params.data=weights.data
However, this would save the weights individually and hence would not achieve any kind of reduction in the model size. I wanted to use and access the same storage (something similar to the following link : PyTorch Views ). I believe to the best of my knowledge that PyTorch does not have any clustering mechanism as yet as opposed to TensorFlow.
I have tried using view in the above code but have not been able to do so successfully getting the desired results.Can someone guide/help me with this?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
