How to Stop the flutter rebuilds from ValueListenableBuilder?

Question

I'm having an issue of the widget subtree constantly rebuilding from inside the ValueListenableBuilder. It is supposed to run a rebuild on change, and in this case it is listening to a table on a Flutter Hive Database.
Things I've tired:

1. I had all of my Hive Boxes open in the main method, so that I have access to each box from anywhere in the app. I tired only opening the Hive box when something is changed, then promptly closing this box. Didn't work

Things I think it could be, but not sure:

1. Mixing ChangeNotifierProvider with the ValueListenableBuilder - Because some of the subtree also utilizes changenotifier, but with ValueListenableBuilder constantly rebuilding the subtree, any changes I pass into the provider get wiped out.

Is there anyway of only rebuilding on a change only?

  @override
  Widget build(BuildContext context) {
    return ValueListenableBuilder(
        valueListenable:
            Hive.box<Manifest>(HiveTables.manifestBox).listenable(),
        child: assignmentWidgets,
        builder: (context, Box<Manifest> manifestBox, child) {
          if (manifestBox.isNotEmpty)
            return child!;
        },
        );
  }