Sort SQL results and include missing keys

Question

I have a Postgres table like this (greatly simplified):

id | object_id (foreign id) | key (text) | value (text)
1 | 1 | A | 0foo
2 | 1 | B | 1bar
3 | 1 | C | 2baz
4 | 1 | D | 3ham
5 | 2 | C | 4sam
6 | 3 | F | 5pam
…
(billions of rows)

I select object_ids according to some query (not relevant here), and then sort them according to the value of a specified key.

def sort_query_result(query, sort_by, limit, offset):
    return query\
        .with_entities(Table.object_id)\
        .filter(Table.key == sort_by)\
        .order_by(desc(Table.value))\
        .limit(limit).offset(offset).subquery()

For example, assume a query matches object_ids 1 and 2 above. When sort_by=C, I want the result to be returned in the order [2, 1], because 4sam > 2baz.

This works well but there's one big problem:

Object ids that are returned by query but do not have any row for the sort_by key, are not returned at all.

For example, for a query that matches object_ids 1 and 2, sort_query_results(query, sort_by='D') == [1]. The object_id 2 is dropped because it has no D, which is undesirable.

Instead, I'd like to return all object_ids from the query. Those without the sort key should be sorted at the end, in any order: sort_query_results(query, sort_by='D') == [1, 2].

What's the best way to achieve that?

---

Note: I do not have the freedom to change the DB schema or business logic. But I can change the query code. I use SQLAlchemy ORM from Python, but could execute raw Postgres commands if necessary. Thank you.