'Fast Load Matrices [Numpy/Java]
Python tries:
I have matrices ~(50000 * 100), and some more of 50*100, (50000*2). I saved them using numpy_compressed and it loads the matrix back in around 300ms. Saving this as Json (list-of-list)[Same as saving to str dump and using eval] takes around 700ms.
My main objective is to load matrices fast for a service which requires fast response. The actual matrix computation, once the matrices have been loaded in memory is ~2ms.
How can the numpy matrices be loaded fast ? Is it faster in Java/ Clojure?
Solution 1:[1]
Your question is about the speed of (de-)serialization. In general, byte/raw serializers are potentially faster than text serialization like JSON. The actual performance will depend on your exact case, so if the performance is really critical, you should test it with different serializers.
This being said, Transit is a good option for transferring data between applications that are written in different languages. On the other hand, a benchmark of Java serializers can be found at: https://github.com/eishay/jvm-serializers/wiki.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Toni Vanhala |