'How can I resolve the memory error while calculating pairwise distance of a high dimensional numpy array and converting into square form?

I have a 16GB RAM laptop. I am using 195159 x 153 data from Home Credit Default Risk | Kaggle. I am trying to calculate pairwise distance of each point and put it in square form.

import scipy as sp
pdist=sp.spatial.distance.pdist(X,metric='sqeuclidean')
Corig = sp.spatial.distance.squareform(pdist,checks=False)

I am getting memory error while calculating spatial.distance.pdist for the entire data error message 1

Tried out on 100K samples, I am able to calculate the pairwise distance (pdist) but getting error while converting into square form error message 2:

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source

'How can I resolve the memory error while calculating pairwise distance of a high dimensional numpy array and converting into square form?

Sources

Related Questions