R selective ggplot geom_point(position = position_dodge())

Question

In the plot below, I'd like to dodge the colored shapes, but not the shape for 1 so that the dotted line would line up with 1. The colored shapes need to be dodged so that they don't overlap one another for being at the same time point. Here's the code for generating dummy data and the plot. Is there a way to selectively dodge points that are in the same geom_point?

df <- data.frame(id = factor(sort(rep(seq(1,5),2))),
                 time = rep(c(3,6), 5),
                 cat1 = c(sample(c('good', 'ok', 'bad'), 2),
                          sample(c('good', 'ok', 'bad'), 2),
                          sample(c('good', 'ok', 'bad'), 2),
                          sample(c('good', 'ok', 'bad'), 2),
                          sample(c('good', 'ok', 'bad'), 2)),
                 cat2 = c(sample(c('a', 'b', 'c', 'd'), 2), 
                          sample(c('a', 'b', 'c', 'd'), 2),
                          sample(c('a', 'b', 'c', 'd'), 2),
                          sample(c('a', 'b', 'c', 'd'), 2),
                          sample(c('a', 'b', 'c', 'd'), 2))) %>%
  pivot_longer(cols = c('cat1', 'cat2'), names_to='type', names_prefix = 'value', values_to = 'value') %>%
  plyr::rbind.fill(data.frame(id = factor(seq(1,5)), 
        time = 9,
        time2 = 9,
        type = 'off',
        value = c(1, NA, NA, 1, 1))) %>%
  dplyr::arrange(id, time)

ggplot(df, aes(x = id, y = time)) +
  geom_point(aes(id, time, colour = value, shape = value), size = 2, position = position_dodge(width = 0.7)) +
  geom_segment(data = df[df$type == 'off',], aes(x = id, xend = id, y = 6,
                                          yend = time2), colour = 'black', linetype = 'dotted') +
  coord_flip() +
  scale_shape_manual(values = c(13, 17, 17, 17, 17, 16, 16, 16, 15, 15, 15, 15)) +
  scale_colour_manual(values = c('black', 'purple', 'green', '#ffff66', 'red',
                                 'green', '#ffff66', 'red', 
                                 'green', '#ffff66', 'pink')) +
  guides(fill = guide_legend(order = 2), shape = guide_legend(override.aes = list(size = 3)))