Three.js - Detect intersection using Orthographic Camera

Question

I am trying to detect the mesh that was clicked on using an orthographic camera.

It works great for objects that are near the center of the viewport. But when objects are near the edges of the viewport, intersection is no longer detected.

renderer.domElement.addEventListener('pointerdown', (event) => {

    let raycaster = new THREE.Raycaster();
    let mouse = new THREE.Vector2();

    mouse.x = ( event.clientX / sizes.width ) * 2 - 1;
    mouse.y = - ( event.clientY / sizes.height ) * 2 + 1;

    raycaster.setFromCamera(mouse, camera);

    var intersects = raycaster.intersectObjects(objectsList, false);
    console.log("Intersects:", intersects);
}

Update 1: My camera is rotated to an isometric-like view; might it have to do with that?

var camera = new THREE.OrthographicCamera(
    sizes.width / -2,
    sizes.width / 2,
    sizes.height / 2,
    sizes.height / -2,
    -100,
    100
);
camera.position.set(16, 0, -16);
camera.rotation.order = 'YXZ';
camera.rotation.y = -30 * THREE.MathUtils.DEG2RAD;
camera.rotation.x = -30 * THREE.MathUtils.DEG2RAD;
camera.zoom = 82;
camera.updateProjectionMatrix();

Update 2: I have noticed that when I change the y coordinate of camera.position to something greater, e.g. 8, it works as expected. Why is that?