'Why does my camera ray have an offset when clicking near the edges of the viewport?

I'm working on a 3D project in OpenGL. Picking objects is my thing to worry about, but right now I can't get the main thing working. Which is casting a ray from my camera using the mouse. I've been digging through too many examples including raycasting without "unproject", but they literally give me the same result! They are almost working, and now I am pretty much stuck and do not know what to do.

Consider my method I've picked last:

function CCamera.GetRay(x: Single; y: Single;) : TVector3;
var
  mouseX: single;
  mouseY: single;
  invVP: TMatrix4;
  screenPos: TVector4;
  worldPos: TVector4;
  worldPos3: TVector3;
  dir: TVector3;
begin

  // NDC
  mouseX := x / (fViewportWidth * 0.5) - 1.0;
  mouseY := y / (fViewportHeight * 0.5) - 1.0;

  invVP := mView * mProjection;
  // invVP.SetInversed; // Inversed doesn't even cast the ray close to where I want it cast at.
  // mProjection * mView would also generate behavior that isn't close to what I want.

  screenPos.Init(mouseX, -mouseY, 1.0, 1.0);
  worldPos := invVP * screenPos;
  worldPos3.Init(worldPos.X, worldPos.Y, worldPos.Z);

  dir := worldPos3.Normalize();

  Exit(dir);
end;

Literally. Every. Single. Example gives me the same result! This is the one I will stick to.

And then, I would determine the camera ray's end position as such:

UPDATED code:

procedure TOpenGLControl.OnMouseDown(Sender: TObject; Button: 
TMouseButton; Shift: TShiftState; X: Integer; Y: Integer);
var
  ray_start: TVector3;
  ray_end: TVector3;
  ray_dir: TVector3;
begin

    ray_start := GetCamera.GetPosition(); // Position of the camera
    ray_dir := GetCamera.GetRay(X, Y); // Pass 2D mouse coordinates
    ray_end := ray_start + ray_dir * 50.0; // Max distance of 50 units

    Invalidate();
end;

Using Neslib.FastMath for the math functions and attributes. So, basically, what happens is that the further I click from the center of the world (where the triangle is), the ray strives closer to the center. Check image.

Solution 1:^[1]

SOLUTION! This is the new unproject method. It is accurate 100%. I managed to put it all together. Weird.

pos.Init(0.0, 0.0, 0.0, 0.0);
pos.X := (x - 0.0) / fViewportWidth * 2.0 - 1.0;
pos.Y := 1 - (y - 0.0) / fViewportHeight * 2.0;
pos.Z := z * 2.0 - 1.0;
pos.W := 1;

mat := mView * mProjection;
mat.SetInversed();

pos2 := pos * mat;

pos_out.Init(pos2.X, pos2.Y, pos2.Z);

Exit(pos_out / pos2.W);

Solution 2:^[2]

Not a Delphi coder but here some insights from BCBs and VCL (should be the same):

use local mouse position (OnMouseMove)
convert mouse position to OpenGL position

beware mouse y axis is in opposite direction to OpenGL and the (0,0) is in center of screen instead of in top left corner. And the range is usually <-1.0,+1.0> From What I see you are not inverting y and have range <0.0,1.0> if I see it right ...

Also beware if you have panels in your Form then you should also check your glViewbox settings as VCL sometimes wrongly calculates ClientWidth/Height and you need to substrack what is missing like - Panel1->Width; ...
beware of you ray origin

in perspective views it has to start from camera focal point. Depending on your projection matrix it might be shifted by znear (focal length).

If not correct this will create bigger error on edges of view so I bet its the cause of your problems.
beware of depth buffer precision

you can tweak perspective so zfar/znear matches your depthbuffer precision.

if not enough or not possible use linear depth buffer

Take a look at these:

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source
Solution 1	IOviSpot
Solution 2

'Why does my camera ray have an offset when clicking near the edges of the viewport?

Solution 1:[1]

Solution 2:[2]

Sources

Related Questions

Solution 1:^[1]

Solution 2:^[2]