How to draw lines connecting pair-wise points above dodging bars in R?

Question

I think the question is sufficiently complex that a code example will help:

library(ggplot2)

df <- data.frame(
  Group = c("A", "A", "A", "A", "B", "B"),
  Subgroup = c("A.1", "A.2", "A.1", "A.2", "B.1", "B.2"),
  Value = c(10, 7, 8, 9, 11, 12),
  Pair = c(1, 1, 2, 2, 3, 3)
)

dodge <- position_dodge(width = 0.9)
ggplot(data = df, mapping = aes(x = Group, y = Value, fill = Subgroup)) +
  geom_bar(stat = "summary", fun = "mean", position = dodge) +
  geom_point(position = dodge) +
  geom_line(color = "red", mapping = aes(group = Pair), position = dodge)

The point is to have a bar chart with groups and subgroups (success), with the individual dots plotted centered above each bar (success), and with lines connecting pairwise samples (fail). The result is not too far off, but apparently, instead of making the points avoid each other and then drawing the lines, ggplot2 draws the vertical lines and then makes them avoid each other.

Actual:

Expected: