Create interactive map-like experience for big SVG file (non-map)

Question

I've been trying to create a smooth map-like experience for exploring a big chart I've made using D3.js

Every node on the chart is linked to a particular webpage and I plan to add info-bubbles to display english translation. You can see some results below. (1) is very laggy when using the D3 function zoom and drag, but gives the appropriate result in terms of interactivity. Using libvips I was able to create custom PNG tiles to experiment with leaflet and openLayer. This works very nicely but have no interactivity at all. I was able to do a small openLayer app to display the entire SVG (6), but the result is laggy.

1. D3 version - SVG - very laggy- (code)
2. Zoomify version - jpeg Tiles - blurry - (code)
3. Leaflet version I - unique PNG - laggy - (code)
4. Leaflet version II - PNG Tiles - smooth - (code)
5. OpenLayer Version I - PNG Tiles - very smooth - (code)
6. OpenLayer Version II - unique SVG canvas ? - very laggy - (code)
7. OpenLayer Version III - unique SVG native - laggy - (code)

I imagine it is possible to transfer the interactive data from the original SVG (I could use <a> element instead of .on("click")) and create a layer on top of the PNG tiles. But I have no idea how to do that. The other alternative I thought was to create SVG tiles or vector tiles, but I could not find anything about that or any library/software that would slice a SVG in tiled SVG files.

SVG mapper sounds promising when I read the description, but seems too specific to geo data to be useful. (Here is their working exemple, although the tiles are missing in the background and zoom is broken)

This leaflet + D3 is interesting also. (with improvement)

Any advice is welcome as I'm a bit stuck at the moment. Thanks !