varchar column is causing distinct order by to be very slow

Question

I have a simple table and query where I am doing:

SELECT DISTINCT ON (security) security, value_date, value_currency, mid_price
    FROM prices
    WHERE value_date <= date '2022-02-18'
    ORDER BY security, value_date DESC,
             CASE value_currency
                 WHEN 'GBP' THEN 3
                 WHEN 'EUR' THEN 2
                 WHEN 'USD' THEN 1
                 ELSE 0
                 end
                 DESC;

This seems to be taking around 4 seconds on about 3.5 million rows. When I remove the value_currency part of the order by then it drops significantly to a third of a second. Even without the janky case statement, this query

SELECT DISTINCT ON (security) security, value_date, value_currency, mid_price
FROM prices
WHERE value_date <= date '2022-02-18'
ORDER BY security, value_currency DESC;

Still takes a few seconds.

I have tried to add an index on value_currency, but that did not seem to make a difference. Struggling to understand what affects the performance of having a varchar in an order by in postgres.