Add geom_point() to cowplot subplot

Question

I have made a cowplot using

gg <- cowplot::plot_grid(p1, p2)

I now want to add a ggplot2::geom_point() layer to p1 but I only have access to gg. How can I pull p1 and p2 back out, add a layer to p1 and then make a new cowplot, cowplot::plot_grid(p1_new, p2)?

I tried

cowplot::plot_grid(ggplot2::ggplot() + 
  gg$layers[[1]]$setup_layer + 
  ggplot2::geom_point(ggplot2::aes(x = x, y = y)), 
  ggplot2::ggplot() + gg$layers[[2]])

but that does not work. I don't really understand how ggplot2 (and cowplot) store the plots, grobs, layers, etc.