Switching keyboard shortcut between multiple buttons in SwiftUI

Question

I have a list of buttons, and would like the one that is currently being focused to be triggered with a shortcut.

I've tried using a conditional .keyboardShortcut modifier, but SwiftUI somehow seems to remember which one it was applied to first, even when the focus state changes.

Also tried using a "fallback key" for the non-focused buttons, so that triggering the desired shortcut affects the focused button.

I suppose you could add the shortcut to the parent of all buttons and figure out which action to trigger based on the focus state, but that seems convoluted and messy.

Simply pressing SPACE on the focused button does not work at all, although I thought it would.

I've tried boil down the issue to a minimal example (using the "fallback key" approach):

      VStack(alignment: .leading, spacing: 6) {
        ItemView(focused: self._focused, tag: 0)
        ItemView(focused: self._focused, tag: 1)
        ItemView(focused: self._focused, tag: 2)
      }

struct ItemView: View {
  @FocusState var focused: Int?
  var tag: Int
  var isFocused: Bool { self.focused == self.tag }
  @State var icon: String = "person.fill"
  
  var body: some View {
    HStack(alignment: .center, spacing: 5) {
      Button(action: { self.icon = self.icon == "person.fill" ? "person.circle" : "person.fill" }) {
        Image(systemName: self.icon)
      }
      .keyboardShortcut(self.isFocused ? .defaultAction : .cancelAction)
      .focusable()
      .focused(self.$focused, equals: self.tag)
//      .font(self.isFocused ? Font.subheadline.weight(.black) : Font.subheadline)
      Text("hello!")
    }
    .padding(3)
  }
}

This seems like a fairly straightforward behaviour and I cannot figure out why nothing works.