Automate manual color selection for `geom_tile`s

Question

I have this type of data:

target_seq <- structure(list(Line = c("130", "131", "132", "133", "134", "135", 
                                      "136", "137", "138", "139", "140", "141", "142"), 
                             Actor = c("R", "R", "R", "R", "B", "R", "B", "B", "B", "M", "M", "M", "M"), 
                             Act_cat = c("ver", "SpeechRec", "ges", "ges", "gaze", "ges", 
                                         "gaze", "gaze", "gaze", "gaze", "gaze", "gaze", "gaze"), 
                             Activity = c("dort drüben könnt ihr sehen wer damals auf der sparrenburg gewohnt hat", 
                                          "schwert", "D-onset", "D-peak", "~", "D-retract", "@tum", 
                                          "~", "@tum", "~", "@tum", "~", "@tum"), 
                             Starttime_ms = c(48825, 48865, 49220, 50080, 50730, 50900, 51009, 51191, 51270, 51486, 51809, 52251, 52333), 
                             Endtime_ms = c(53035, 49865, 50080, 50900, 51009, 52220, 51191, 51270, 53474, 51808, 52250, 52332, 53226), 
                             Duration = c(4210, 1000, 860, 820,279, 1320, 182, 79, 2204, 322, 441, 81, 893), 
                             File = c("VP_4_004", "VP_4_004", "VP_4_004", "VP_4_004", "VP_4_004", "VP_4_004", 
                                      "VP_4_004", "VP_4_004", "VP_4_004", "VP_4_004", "VP_4_004", 
                                      "VP_4_004", "VP_4_004"), 
                             Action = c("R_ver", "R_SpeechRec", "R_ges", "R_ges", "B_gaze", "R_ges", "B_gaze", "B_gaze", 
                                          "B_gaze", "M_gaze", "M_gaze", "M_gaze", "M_gaze")), 
                        class = c("tbl_df","tbl", "data.frame"), row.names = c(NA, -13L))

I want to plot all Activity values as geom_tiles. I can do that with this code:

library(ggplot2)
ggplot(target_seq,
       aes(y = Action)) +
  # draw tiles for each `Activity` (width of tiles determined by `Duration`):
  geom_tile(aes(x = Starttime_ms + Duration/2, width = Duration, height = 0.3, fill = Actor), color = "white") +
  # print text of `Activity`s:
  geom_text(aes(x = Starttime_ms, label = Activity), hjust = 0, size = 3.5) +
  # define labels for x- and y-axis and plot tile:
  labs(
    # x- axis label:
    x = "Activity duration",
    # y-axis label:
    y = "Actors and activity type",
    # plot title:
    title = paste0("Activities during robot's utterance\nFile: ", 
                   target_seq$File, 
                   " [", 
                   min(target_seq$Line, na.rm = TRUE),
                   " - ",
                   max(target_seq$Line, na.rm = TRUE),
                   "]"))

which gives:

I'd like to choose my own colors for the tiles. To complicate matters a little, this target_seq is just one out of many and the number and identity of the Actors changes from File to File. How can I automate a manual selection of colors for the geom_tiles?