'Large Matrix Multiplication using PDL - Perl
I need to generate (40000 x 20000) matrix via SQL and perform matrix operation like A'A. Then I need to obtain Eigen value and Eigen Vectors.
Can you suggest me ways to implement this in Perl.
(Maybe via PDL)
How can I multiply large matrix stored in file.?
Thank You
Solution 1:[1]
Using the latest PDL, you can assemble various pieces to achieve what you're aiming for:
- https://metacpan.org/pod/PDL::IO::FastRaw#mapfraw to memory-map a disk file as a PDL ndarray, with no RAM limitations
- https://metacpan.org/pod/PDL::IO::DBI to read from a DBI handle
- https://metacpan.org/pod/PDL::LinearAlgebra to use LAPACK routines for various fast matrix operations including eigenvalues - make sure you either turn off PDL auto-parallelism if you use a parallel LAPACK such as OpenBLAS
If David is right and you are actually aiming to do least-squares fitting, PDL::LinearAlgebra has routines for that as well.
Solution 2:[2]
$PDL::BIGPDL=1; $c = sequence(60000, 60000); ## you need 32gb of ram and about 75gb of swap to do this.
$c * $d is different then $c x $d in PDL I believe there are modules for Eigen value or Eigen vector I have not used them yet tho...
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Ed. |
| Solution 2 |