Add legend for different colors of fill_between() segments

Question

I am creating an "event plot" that is currently looking like this:

However, I don't know how I can add a legend just for each color group. This is how the plot gets created at the moment:

handles = dict()

for i, channel_events in enumerate(channel_event_list):

    for event in channel_events:
        start = event[0]
        end = event[1]
        y = (i, i + padding)
        
        c = 'red' if colors is None else colors[i]

        h = plt.fill_between([start, end], y[0], y2=y[1], color=c)
        
        if c not in handles:
            handles[c] = list()
        handles[c].append(h)

I thought that I can use the output of fill_between() as handles but it seems I was mistaken there.

So what would be the easiest way to get a legend only for the colors here?