geom_abline() forces y-axis to 0

Question

Is there a way to add geom_abline() without changing the scales? I've done a bit of digging and haven't found anything that addresses this specifically, and manually rescaling the axes removes the abline:

library(tidyverse)

# generate noise that should be roughly 1:1
plot <- 
  runif(100, 3000, 6000) %>%
  as_tibble() %>%
  rename(x = value) %>%
  rowwise() %>%
  mutate(y = rnorm(1, x, 150)) %>%
  ggplot(aes(x, y)) +
  geom_point()

plot

# geom_abline forces y-axis to 0!
plot +
  geom_abline()

# adding in manual scales removes geom_abline
plot +
  geom_abline() +
  scale_y_continuous(limits = c(3000, 6000))
#> Warning: Removed 1 rows containing missing values (geom_abline).

# reording when `scale_y_continous()` is called doesn't help unfortunately
plot +
  scale_y_continuous(limits = c(3000, 6000)) +
  geom_abline()
#> Warning: Removed 1 rows containing missing values (geom_abline).

packageVersion("ggplot2")
#> [1] '3.3.4'

^{Created on 2022-03-04 by the reprex package (v2.0.1)}