How to decode large json array in flutter

Question

I have a rather large json file that I have downloaded using my flutter app from a web api that's around 200MB using

final response = Dio().get(uri, options: Options(responseType: ResponseType.plain));

(Using default Dio options also uses json decoding, resulting in an out of memory exception. This is not directly part of my problem but maybe it helps)

Now the problem starts after obtaining the json string. When I use jsonDecode on the response body I run out of memory. (maybe the same function Dio uses?)

final data = jsonDecode(response.body); this runs out of memory

The json object itself is an array with a lot of items with a format like this:

[
  {"data": {independent data1}},
  {"data": {independent data2}},
  ...
]

and i would be fine just decoding one item at a time instead of everything at once to reduce memory usage. Is there another way to do process all the items in this array? Something like

jsonArrayDecode(response.body, onItemDecode: (item) { /*do stuff with item */ });

Or do I have to write my own json reader that can sequentially decode it?

Answer 1

Thanks to @pskink, I managed to solve my problem. I used the ResponseType.stream from Dio to get stream which can be processed using the reviver function argument from the JsonDecoder. I process all the data in the reviver and ignore the data in the onData function of the listener.

...
final response = await dio.get(uri,
        options: Options(responseType: ResponseType.stream);
Function reviver = (key, value) {
  if (/*value is element from top level list*/) {
    processItem(value);
    return null;
  }
  return value;
}
final completer = Completer();
response.data.stream.cast<List<int>>()
        .transform(utf8.decoder)
        .transform(JsonDecoder(reviver))
        .listen(null,  onDone: () { completer.complete(); });
await completer.future;
...

Answer 2

If the problem is really that the data file is very large, then the problem can be solved by not storing unnecessary objects during parsing.
This will only make sense if not all of the data is needed, but only some of the data.
In this case, we are talking about filtering data on the fly, that is, directly during the data parsing process.

Below is a small example of how this can be implemented.

import 'package:fast_json/fast_json_selector.dart' as parser;
import 'package:fast_json/fast_json_selector.dart' show JsonSelectorEvent;

void main(List<String> args) {
  // Select users from the list by indexes [2..3] and terminate selection
  final users = <User>[];
  final level = '{} data [] 0 {}'.split(' ').length;
  final elementLevel = '{} data [] 0'.split(' ').length;
  void select(JsonSelectorEvent event) {
    final levels = event.levels;
    if (levels.length == level) {
      final index = event.levels[elementLevel - 1] as int;
      if (index >= 2 && index <= 3) {
        final map = event.lastValue as Map;
        final user = User.fromJson(map);
        users.add(user);
      }

      // Free up memory
      event.lastValue = null;
      if (users.length == 2) {
        throw const _TerminateException();
      }
    }
  }

  try {
    parser.parse(_data, select: select);
  } on _TerminateException {
    //
  }

  print(users.join(', '));
}

const _data = '''
{
   "success":true,
   "data":[
      {
         "id":1,
         "name":"Leanne Graham",
         "username":"Bret",
         "email":"[email protected]",
         "address":{
            "street":"Kulas Light",
            "suite":"Apt. 556",
            "city":"Gwenborough",
            "zipcode":"92998-3874",
            "geo":{
               "lat":"-37.3159",
               "lng":"81.1496"
            }
         },
         "phone":"1-770-736-8031 x56442",
         "website":"hildegard.org",
         "company":{
            "name":"Romaguera-Crona",
            "catchPhrase":"Multi-layered client-server neural-net",
            "bs":"harness real-time e-markets"
         }
      },
      {
         "id":2,
         "name":"Ervin Howell",
         "username":"Antonette",
         "email":"[email protected]",
         "address":{
            "street":"Victor Plains",
            "suite":"Suite 879",
            "city":"Wisokyburgh",
            "zipcode":"90566-7771",
            "geo":{
               "lat":"-43.9509",
               "lng":"-34.4618"
            }
         },
         "phone":"010-692-6593 x09125",
         "website":"anastasia.net",
         "company":{
            "name":"Deckow-Crist",
            "catchPhrase":"Proactive didactic contingency",
            "bs":"synergize scalable supply-chains"
         }
      },
      {
         "id":3,
         "name":"Clementine Bauch",
         "username":"Samantha",
         "email":"[email protected]",
         "address":{
            "street":"Douglas Extension",
            "suite":"Suite 847",
            "city":"McKenziehaven",
            "zipcode":"59590-4157",
            "geo":{
               "lat":"-68.6102",
               "lng":"-47.0653"
            }
         },
         "phone":"1-463-123-4447",
         "website":"ramiro.info",
         "company":{
            "name":"Romaguera-Jacobson",
            "catchPhrase":"Face to face bifurcated interface",
            "bs":"e-enable strategic applications"
         }
      },
      {
         "id":4,
         "name":"Patricia Lebsack",
         "username":"Karianne",
         "email":"[email protected]",
         "address":{
            "street":"Hoeger Mall",
            "suite":"Apt. 692",
            "city":"South Elvis",
            "zipcode":"53919-4257",
            "geo":{
               "lat":"29.4572",
               "lng":"-164.2990"
            }
         },
         "phone":"493-170-9623 x156",
         "website":"kale.biz",
         "company":{
            "name":"Robel-Corkery",
            "catchPhrase":"Multi-tiered zero tolerance productivity",
            "bs":"transition cutting-edge web services"
         }
      },
      {
         "id":5,
         "name":"Chelsey Dietrich",
         "username":"Kamren",
         "email":"[email protected]",
         "address":{
            "street":"Skiles Walks",
            "suite":"Suite 351",
            "city":"Roscoeview",
            "zipcode":"33263",
            "geo":{
               "lat":"-31.8129",
               "lng":"62.5342"
            }
         },
         "phone":"(254)954-1289",
         "website":"demarco.info",
         "company":{
            "name":"Keebler LLC",
            "catchPhrase":"User-centric fault-tolerant solution",
            "bs":"revolutionize end-to-end systems"
         }
      }
   ]
}''';

class Company {
  final String name;

  Company({required this.name});

  @override
  String toString() {
    return name;
  }

  static Company fromJson(Map json) {
    return Company(
      name: json['name'] as String,
    );
  }
}

class User {
  final String city;
  final int id;
  final String name;

  User({required this.city, required this.id, required this.name});

  @override
  String toString() {
    return '$id:$name from $city';
  }

  static User fromJson(Map json) {
    return User(
      city: json['address']['city'] as String,
      id: json['id'] as int,
      name: json['name'] as String,
    );
  }
}

class _TerminateException {
  const _TerminateException();
}