Restrict host groups to load into memory for playbook execution

Question

I have a playbook that has hundreds of tasks which updates configuration, package deployment and validations. Based on my understanding all the host groups within the inventory file is getting loaded into memory during gathering facts which is not needed. Not just gathering facts but the entire playbook execution is delayed and I suspect its due to loading of all host groups into memory at the beginning of playbook.

There are around 5k hosts in inventory file but I am mainly focused on the target hostgroup to be considered for task execution which hardly has 110 hosts. Is there a way to restrict ansible not to consider other hostgroups to load into memory and just ignore since the host or hostgroup is not matching that is passed for playbook using limit option.

Since the bastion host run multiple playbook execution happens in parallel we are seeing considerable delay in gathering facts and also task execution is delayed. Any suggestion how to restrict loading of entire inventory file into memory will be appreciated.

Thanks