GitHub Actions - Cache CMake build

Question

On every commit, I'm building my project on different Operating Systems. Therefore I'm using CMake as a build system for my C++ code. In order to make the builds faster I tried to cache older builds, so not changed files don't have to be rebuilt. I've written the following GH Action Script:

name: Build for MacOS, Ubuntu and Windows

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

env:
  BUILD_TYPE: Release

jobs:
  build:
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        os: [ ubuntu-20.04, macos-11, windows-2022 ]
    steps:
      - uses: actions/checkout@v2

      - name: Cache build
        uses: actions/cache@v3
        with:
          path: ${{github.workspace}}/build
          key: ${{ matrix.os }}-build
          restore-keys: ${{ matrix.os }}-build

      - name: Configure CMake
        run: cmake -B ${{github.workspace}}/build -DCMAKE_BUILD_TYPE=${{env.BUILD_TYPE}} -DFORCE_COLORED_OUTPUT=1

      - name: Build
        run: cmake --build ${{github.workspace}}/build --config ${{env.BUILD_TYPE}}

Width ...

- name: Cache build
        uses: actions/cache@v3
        with:
          path: ${{github.workspace}}/build
          key: ${{ matrix.os }}-build
          restore-keys: ${{ matrix.os }}-build

... I tried to cache the directory all the build files are written to, but it looks like the project is completely rebuilt every time. Is there anything I'm doing wrong? I uploaded all logs to https://pastebin.com/JLErAPyD