Docker container cannot connect to other machines or clone from gitlab using ssh (Start Rundeck with non-root User)

Question

I'm trying to use Rundeck inside a docker container. Starting Rundeck works. Bit I cannot ssh into other machines. I think this is because Rundeck is running with root user. When I connect into my container I'm the root user. Connection via ssh fails. When I switch to user "rundeck" I can connect to another machine (tested this with a connection to one of my RasPi). I confirmed by using whoami inside a simple Rundeck job, that Rundeck is running with root as well.

My Dockerfile for my Rundeck image:

FROM rundeck/rundeck:3.4.10
LABEL maintainer="[email protected]"

USER root
RUN apt-get update \
    && apt-get install -y git

USER "${uid}:${gid}"
RUN mkdir -p /home/rundeck/repos

ENV SERVER_URL=http://localhost:7440 \
    RUNDECK_GRAILS_URL=http://localhost:7440 \
    RUNDECK_SERVER_ADDRESS=0.0.0.0

And my docker-compose.yml:

---
version: "3.3"
services:

  # Schedule tasks and jobs
  rundeck:
    image: pegasus/rundeck:latest
    container_name: rundeck
    command: -Dserver.http.port=7440
    tty: true
    volumes:
      - rundeck_data:/home/rundeck/server/data
      - ./assets/rundeck/inventory.yml:/home/rundeck/inventory.yml
      - /home/vagrant/.gitconfig:/home/rundeck/.gitconfig:ro
      - /home/vagrant/.ssh:/home/rundeck/.ssh:ro
    ports:
      - 7440:7440

volumes:
  rundeck_data:

Cloning from gitlab does not work as well because root user-does not recognize any og my .ssh files.