Adding some value on top of stacked barplot - Bars disappear when defining aes(y) in geom_text

Question

I want to produce a stacked barplot and the percentage change between the years on top of the stacked bars.

My data:

my_data <- structure(list(year = c(2016, 2016, 2016, 2016, 2017, 2017, 2017, 
2017, 2018, 2018, 2018, 2018, 2019, 2019, 2019, 2019, 2020, 2020, 
2020, 2020, 2021, 2021, 2021, 2021), some_data = c(700.884929, 
123.61278, 91.331399, 5.800863, 709.664147, 5.255171, 125.568917, 
94.289424, 721.353624, 2.440006, 130.295319, 96.372448, 726.963262, 
0.875374, 129.695907, 97.54507, 764.370595, 0.876399, 126.677571, 
105.180633, 801.974246, 0.787544, 131.254017, 109.521045), some_letters = c("B", 
"A", "D", "C", "B", "C", "A", "D", "B", "C", "A", "D", "B", "C", 
"A", "D", "B", "C", "A", "D", "B", "C", "A", "D")), row.names = c(NA, 
-24L), class = c("tbl_df", "tbl", "data.frame"))

Percentage change information:

for_label <- structure(list(year = c(2016, 2017, 2018, 2019, 2020, 2021), 
    pct_change = c(NA, 0.0142656905848388, 0.0167780411191876, 
    0.00485892011456412, 0.0440021799523009, 0.0465664546661004
    ), my_pos = c(921629971, 934777659, 950461397, 955079613, 
    997105198, 1043536852)), row.names = c(NA, -6L), class = c("tbl_df", 
"tbl", "data.frame"))

Now I try to plot the data and add the percentages:

library(tidyverse)

my_data %>% 
  ggplot(aes(x = year, y = some_data, fill = some_letters))+
  geom_bar(position = "stack", stat = "identity") +
  scale_x_continuous(breaks = 2016:2021)+
  geom_text(
    data = for_label,
    aes(x = year, y = pct_change, label = percent(pct_change), fill = NULL), 
    position = "stack")

However, the values are on on the bottom of the plot:

Alternatively I change the pct_change to my_pos in geom_text(aes(y = )). Then I get the percentages at the right position but the bars disappear. How to solve this problem? I tried to play around with position = position_stack() but never got it to work.