How to perform an upsert on a table with both a primary key and UNIQUE column

Question

New to SQLite and trying to understand the upsert functionality.

I have a table with the following DDL:

CREATE TABLE contacts (
    contact_id INTEGER PRIMARY KEY,
    first_name TEXT NOT NULL,
    last_name TEXT NOT NULL,
    email TEXT NOT NULL UNIQUE,
    phone TEXT NOT NULL UNIQUE
);

Let's say I insert a record:

INSERT INTO contacts (contact_id, first_name, last_name, email, phone)
VALUES (1, 'John', 'Jones', '[email protected]', '888-867-5309');

How can I do an upsert that takes into account both the UNIQUE constraint (email) and the PK constraint ( contact_id) so that it handles either case since I don't know which constraint will fail.

I tried doing this:

INSERT INTO contacts (contact_id, first_name, last_name, email, phone)
VALUES (1, 'John', 'Jones', '[email protected]', '888-867-5309')
ON CONFLICT (contact_id, email) DO UPDATE
SET first_name='John', last_name='Jones', email='[email protected]', phone='888-867-5309'
WHERE contact_id=1;

But I get the error:

sqlite3.OperationalError: ON CONFLICT clause does not match any PRIMARY KEY or UNIQUE constraint

Doing them individually works just fine.

INSERT INTO contacts (contact_id, first_name, last_name, email, phone)
VALUES (1, 'John', 'Jones', '[email protected]', '888-867-5309')
ON CONFLICT (contact_id) DO UPDATE
SET first_name='John', last_name='Jones', email='[email protected]', phone='888-867-5309'
WHERE contact_id=1;

INSERT INTO contacts (contact_id, first_name, last_name, email, phone)
VALUES (1, 'John', 'Jones', '[email protected]', '888-867-5309')
ON CONFLICT (email) DO UPDATE
SET first_name='John', last_name='Jones', email='[email protected]', phone='888-867-5309'
WHERE email='[email protected]';

I understand I get the error because the combination of columns doesn't meet a single constraint, it encompasses two. But how would I take both into account?