Typescript How to return different types in a function with conditional type

Question

Let's say we've got three domain models:

export interface Invoice {
  readonly id: string;
}

export interface Contract {
  readonly id: string;
}

export interface Project {
  readonly id: string;
  // readonly invoices?: Invoice[];
  // readonly contracts?: Contract[];
}

Now we want to write a method to load a project (with or without its invoices and/or contracts), like a repository:

function load(options): Project

I'm trying to use conditional types to check if invoices and/or contracts are loaded:

type With = {
  readonly invoices?: true;
  readonly contracts?: true;
};

type LoadingOptions = {
  readonly with?: With;
};

function load<O extends LoadingOptions >(options: O): Project &
  (O["with"] extends With & { invoices: true } ? { invoices: Invoice[] } : {}) &
  (O["with"] extends With & { contracts: true } ? { contracts: Contract[] } : {}) {
  // load and return the project with or without its invoices and/or contracts.
}

The function signature seems to work just fine:

type ProjectWithInvoices = Project & { invoices: Invoice[] };
type ProjectWithContracts = Project & { contracts: Contract[] };
type ProjectWithEverything = ProjectWithInvoices & ProjectWithContracts;

// Every one is happy here :)
const prj1: Project = findOne({});
const prj2: ProjectWithInvoices = findOne({ with: { invoices: true } });
const prj3: ProjectWithContracts = findOne({ with: { contracts: true } });

Now the challenge is how to actually build the return value and return it. I can't even make it work with the simple example below:

function load<O extends LoadingOptions>(options: O): Project &
  (O["with"] extends With & { invoices: true } ? { invoices: Invoice[] } : {}) &
  (O["with"] extends With & { contracts: true } ? { contracts: Contract[] } : {}) {
  const project = {
    id: "project 1",
  };

  const invoices = options.with?.invoices ? [{ id: "invoice 1" }] : undefined;
  const contracts = options.with?.contracts ? [{ id: "contract 1" }] : undefined;

  // TSC is not happy with this one!
  return {
    ...project,
    invoices: invoices,
    contracts: contracts,
  };
}

Is there any way to make this work? Please note obviously the actual implementation is not gonna be returning a hard coded value.
I found a couple of similar question on SO, but none seems similar to what I'm trying to write. Thanks.