Storing different types in a single Hashmap

Question

A bit of context to prevent xy:

I want to build a cache for my system. There are Type pairs consisting of IdType and EndpointType in a 1:1 relationship. Every IdType only ever refers to one EndpointType and vice versa. The Id trait (see code below) is not required, it only exists in my current iteration of trying to make this work.

There are way too many different types of endpoints - and usually an application will only use a small subset of them - to justify statically building a cache per endpoint and holding that in memory. Additionally, I want to be able to add new endpoints without touching any code related to caching or the client itself.

The Endpoint trait objects are not object safe and there is no way to make them object safe due to associated consts, Sized and methods not using self to make use of compile time optimizations.

I came up with the idea of storing them as kind of Any type. Now I like type safety, so I tried to restrict it a bit more. After failing to find a satisfying solution, I still have several issues with my idea:

1. How do I make this approach work?
2. Is there a better solution?
3. Is this sound? Why?
4. Is there a way to make it sound?
5. Can I achieve this without unsafe?

Link to the rust playground

use std::sync::Mutex;
use std::collections::HashMap;

pub trait Endpoint: Sized {
    type Id;
}

pub trait Id {
    type Endpoint;
}

pub struct Client {
    cache: Mutex<Cache>,
}

impl Client {
    fn get<T: Endpoint>(&self, id: T::Id) -> T {
        if let Some(result) = self.cache.lock().unwrap().get(id) {
            return result;
        }
        todo!()
    }
}

pub struct Cache {
    map: HashMap<Box<dyn Id>, Box<dyn Endpoint>>,
}

impl Cache {
    fn get<T: Id>(&self, id: T) -> Option<T::Endpoint> {
        if let Some(endpoint) = self.map.get(Box::new(id)) {
            let endpoint: Box<T::Endpoint> = unsafe { std::mem::transmute(endpoint) };
            Some(endpoint)
        } else {
            None
        }
    }
}