'Dot product of two columns containing sparse vectors

I have a pyspark dataframe with two columns containing sparse vectors. I want to create a third column (Column C) containing the element-wise product of these two column

Column A:
(262144,[45252,99197,108625],[1.0,2.0,1.0])
Column B
(262144,[45252,99197,108625],[1.252762968495368,1.6945957207744073,1.252762968495368])

I'm looking to get:

Column C
(262144,[45252,99197,108625],[1/1.252762968495368,2/1.6945957207744073,1/1.252762968495368])

How can I do it in the most scalable way? Thank you.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source