How to handle Vue 2 memory usage for large data (~50 000 objects)

Question

I'm trying to implement an table-view for large collections of semi-complex objects on Vue 2. Basically the idea is to collect anywhere between 50 000 to 100 000 rows from DB into JS cache, which is then analyzed dynamically to build table-view with real-time-filters (text-search). Each row within table is toggleable, meaning that clicking the row changes the row to edit-mode, which enables Excel-like editing for that specific field/cell.

Each object has about ~100-150 fields/properties, but only certain amount of 'em are shown at any given moment within table (table columns can be toggled in real-time). For large datasets it seems that DB is pushing about ~10-100mb of JSON-data, which in this use-case is acceptable. Renderwise the performance isn't an issue -- filters work fast enough and only limited amount of results are rendered to DOM.

Everything already works, filters, listing ~100 rows against filters (+ "show 100 more"-mechanism etc), but I hit memory limit when I loaded around 8000 objects into array. This seems to reserve 2 gigabytes of RAM, which after Chrome stops running JS-code all together (even though strangely I'm not getting any kind of warning/error).

I benchmarked memory usage for rows and it seems that ~1000 rows reserves around 300mb of memory. This is most likely reserved by Vue reactivity observers.

Three questions:

1. Is there a way to toggle reactivity for specific array-list objects (by index or such), so that objects within array itself are unobserved/non-mutable unless specifically called to become mutable (ie. when user clicks row, which enables edit-mode)?
2. How would you implement handling of large datasets for Vue, as reactivity seems to bottleneck the memory usage? Please do not suggest "limit the results within backend", because it's not the solution which I'm seeking here (even though I may need to create two-part filtering, one for fetching smaller initial dataset and one for realtime filtering). Basically I'm trying to push boundaries of "end of memory" from 8 000 -> 80 000 by re-thinking the data-architecture with Vue. Is the only problem having dataset storaged within Vue's data-variables as reactive?
3. One idea I have is to turn that "items" -dataset to non-observable/non-reactive with Object.freeze or some similar approach and have table to render two datasets: one for non-reactive and one for those which are currently within edit-mode (which would be pushed to "editableItems" dataset when row is clicked)... any suggestions here (anything simpler, so that I'm able to handle everything within one array?)

I have done similar application on Angular 1, and it handled 50 000 rows quite well, so I'm sure it should be doable within Vue 2 as well... should be just a matter of finding a way on handling reactivity.

Answer 1

From everything I've read, I see that you just don't need reactivity for that data, because:

Each row within table is toggleable, meaning that clicking the row changes the row to edit-mode, which enables Excel-like editing for that specific field/cell

This means rows are not editable and data cannot be mutated without user interaction.

Each object has about ~100-150 fields/properties, but only certain amount of 'em are shown at any given moment within table (table columns can be toggled in real-time).

You keep fields reactive but not display them.

---

And now your questions

Is there a way to toggle reactivity for specific array-list objects (by index or such), so that objects within array itself are unobserved/non-mutable unless specifically called to become mutable (ie. when user clicks row, which enables edit-mode)?

If there's a single item that can be edited at a time, then why keep everything reactive? You can easily use a single variable to listen for that changes.

How would you implement handling of large datasets for Vue, as reactivity seems to bottleneck the memory usage?

It's all about implementation - you rarely end up in a situation when you need a huge list of items to be reactive. The more items you have, the more events needs to happen in order to use the reactivity. If you have 50k items and there are just a few events to mutate (like user modifying data manually), then you can easily listen for those events and make the reactivity manually rather than leave Vue handle all the data. You can check Vuex that can make your life a bit easier for you :)

One idea I have is to turn that "items" -dataset to non-observable/non-reactive with Object.freeze or some similar approach and have table to render two datasets: one for non-reactive and one for those which are currently within edit-mode (which would be pushed to "editableItems" dataset when row is clicked)

This is kind of going in the right direction but there is no need to support two arrays. Imagine using something like this:

data: function() {
    return {
        editingItem: {}
        // when editing is enabled bind the input fields to this item
    }
},
created: function() {
    this.items = [] // your items, can be used in markdown in the loop, but won't be reactive!
},
watch: {
    editingItem: function(data) {
        // this method will be called whenever user edits the input fields
        // here you can do whatever you want
        // like get item's id, find it in the array and update it's properties
        // something like manual reactivity ;)
    }
}

Answer 2

• I had this exact problem where I needed to display a huge list, think 50000 items at least of variable height and I could not find any solution for it
• The general solution is to build/use a virtual scroll.
• It only keeps a few items in DOM while the rest of them are edited in the DOM. It however keeps changing what is visible depending on whether you scroll up/down
• The existing libraries I find do not deal with dynamic heights unless you HARDCODE the heights like vue-virtual-scroller and vue-virtual-scroll-list
• vue-collection-cluster allows you to dynamically calculate heights but lags miserably at 50000 items
• So I came up with my own solution that scrolls SUPER SMOOTH at 50000+ items, even tested with 100k items and works pretty well
• The idea of the implementation for dynamic row heights goes like this

• We need to maintain a list of heights for each item in an array

• Based on where the scroll Top is we apply a transform translateY vertically to offset the few items that we show the user at all times

• I have added ENOUGH comments in the solution for you to easily figure out what is going on

HTML

<script type="text/x-template" id="virtual-list">
   <div id="root" ref="root">
      <div id="viewport" ref="viewport" :style="viewportStyle">
        <div id="spacer" ref="spacer" :style="spacerStyle">
         <div v-for="i in visibleItems" :key="i.id" class="list-item" :ref="i.id" :data-index="i.index" @click="select(i.index)"  :class="i.index === selectedIndex ? 'selected': ''">
           <div>{{ i.index + ' ' + i.value }}</div>
   </div>
   </div>
   </div>
   </div>
</script>
<div id="app">
   <h1 class="title">
      Vue.js Virtual + Infinite Scroll + Dynamic Row Heights + Arrow Key Navigation + No Libraries
   </h1>
   <p class="subtitle">
      No hardcoding of heights necessary for each row. Set emitEnabled to false
      for max performance. Tested with <span id="large_num">50000</span> items...
   </p>
   <div id="list_detail">
      <div id="list">
         <virtual-list></virtual-list>
      </div>
      <div id="detail">
         <table>
            <tbody>
               <tr>
                  <th class="caption">Root Container Height</th>
                  <td>{{store['root-height']}} px</td>
               </tr>
               <tr>
                  <th class="caption">Viewport Height</th>
                  <td>{{store['viewport-height']}} px</td>
               </tr>
               <tr>
                  <th class="caption">Smallest Row Height</th>
                  <td>{{store['smallest-height']}} px</td>
               </tr>
               <tr>
                  <th class="caption">Largest Row Height</th>
                  <td>{{store['largest-height']}} px</td>
               </tr>
               <tr>
                  <th class="caption">Scroll Top</th>
                  <td>{{store['scroll-top']}} px</td>
               </tr>
               <tr>
                  <th class="caption">Page Index</th>
                  <td>{{store['page-start-index']}}</td>
               </tr>
               <tr>
                  <th class="caption">Start Index</th>
                  <td>{{store['start-index']}}</td>
               </tr>
               <tr>
                  <th class="caption">End Index</th>
                  <td>{{store['end-index']}}</td>
               </tr>
               <tr>
                  <th class="caption">Translate Y</th>
                  <td>{{store['translate-y']}} px</td>
               </tr>
            </tbody>
         </table>
         <p><b>Visible Item Indices on DOM</b> {{store['visible-items']}}</p>
         <p><b>Total Height Till Current Page</b> {{store['page-positions']}}</p>
         <p>
            <b>Row's Vertical Displacement From Viewport Top</b>
            {{store['row-positions']}}
         </p>
         <p><b>Heights</b> {{store['heights']}}</p>
      </div>
   </div>
</div>

CSS

@import url('https://fonts.googleapis.com/css?family=Open+Sans&display=swap');

* {
  margin: 0;
  padding: 0;
  box-sizing: border-box;
}


/**
Apply Scroll Bar Styles

https://css-tricks.com/the-current-state-of-styling-scrollbars/
*/
html {
  --scrollbarBG: #181C25;
  --thumbBG: orange;
}
body::-webkit-scrollbar {
  width: 11px;
}
body {
  scrollbar-width: thin;
  scrollbar-color: var(--thumbBG) var(--scrollbarBG);
}
body::-webkit-scrollbar-track {
  background: var(--scrollbarBG);
}
body::-webkit-scrollbar-thumb {
  background-color: var(--thumbBG) ;
  border-radius: 6px;
  border: 3px solid var(--scrollbarBG);
}


html {
  height: 100%;
}

body {
  min-height: 100%;
  height: 100%;
  padding: 2rem;
  color: #AAA;
  background: #181C25;
  font-family: 'Open Sans', sans-serif;
  font-size: 0.9rem;
  line-height: 1.75;
}

#app {
  height: 100%;
  display: flex;
  flex-direction: column;
}

#list_detail {
  display: flex;
  height: 70%;
}

#list {
  flex: 2;
  height: 100%;
}

#detail {
  flex: 1;
  padding: 1rem;
  overflow: auto;
  height: 100%;
}

#root {
  height: 100%;
  overflow: auto;
}

.list-item {
  padding: 0.75rem 0.25rem;
  border-bottom: 1px solid rgba(255, 255, 0, 0.4);
}

.title {
  color: white;
  text-align: center;
}

.subtitle {
  color: orange;
  text-align: center;
}

table {
  width: 100%;
  table-layout: fixed;
  text-align: center;
}

th.caption {
  text-align: left;
  color: #00BEF4;
  font-weight: 100;
  padding: 0.5rem 0;
}

td {
  text-align: left;
}

b{
  font-weight: 100;
  color: #00BEF4;
}

#large_num {
  color: red;
}

.selected {
  background: midnightblue;
}

Vue.js

I am getting limited to 30000 characters here on SO and therefore HERE is the complete code on CodePen

Limitations

• Does not play nice with screen resize at the moment, working on it

Features

• 50000+ items effortless scroll

• Arrow navigation supported just like a native list

• If you have any questions, let me know in the comments

Answer 3

To render large list of components I just found https://github.com/RadKod/v-lazy-component really love it. It just use the Intersection API to render, or not, the component. It removes the non-visible lazy loaded components and load the visible ones in a very smooth way with no innecesary complications.