Is there a way to show the significance of category means against a set value, not one another?

Question

I am graphing measured results versus expected results from a model, grouped by categories (the category in the boxplot below is one of a few different ones I'm using). For each data point, I subtracted the expected from the observed to determine the difference. My task is to modify the model to minimize the difference.

I would like to add the significance level to this chart but all resources I am finding are to compare means of each category to one another. In this case, I would like to know if each of the category's means is significantly different from 0. I can run this test one by one, selecting for data points falling within each category and testing for a difference from 0, but this seems inefficient.

Is there a way to automatically generate this and plot it? stat_compare_means seemed promising but I couldn't figure out how to make it work, while stat_pvalue_manual may hold more promise if I figure out how to code this.

Thanks in advance!

Sample boxplot (too new to add preview)