How to make the enemy look at the player if the player is not hiding behind a wall?

Question

The enemy must look at the player if the player comes into view. Below is the code how I did it. The problem is that the enemy sees the player through the wall. How can I fix the code so that the enemy does not see the player through obstacles?

I came up with this solution - to filter out all hits to the player, and then additionally filter out those that do not pass through the barrier. However, I don't know how to implement it.

RaycastHit[] hitsInfo = Physics.SphereCastAll(head.position, sightDistance, transform.forward, sightDistance);

for (int i = 0; i < hitsInfo.Length; i++)
  if (hitsInfo[i].transform.gameObject.tag == "Player") {
    transform.LookAt(hitsInfo[i].transform.position);
    transform.eulerAngles = new Vector3(0, transform.eulerAngles.y, 0);

    Debug.DrawLine(head.position, hitsInfo[i].transform.position, Color.red, 0.05f, true);
    break;
  }

Below is an illustration: the player is white, the enemy is blue, the red is a raycast visualization.

Answer 1

You can use math to check whether the player in sight or not and than use raycast to filter all obstacles.

Here is an example:

using UnityEngine;

public class Enemy : MonoBehaviour
{
    [SerializeField] private Transform _objectOfInterest;
    [SerializeField] private float _sightAngle = 60f;
    [SerializeField] private float _maxDistance = 10f;

    private void Update()
    {
        if (InSight(_objectOfInterest))
        {
            Debug.Log("InSight");
            if (BehindObstacle((_objectOfInterest)))
            {
                Debug.Log("BehindObstacle");
            }
        }
    }

    private bool InSight(Transform objectOfInterest)
    {
        var forwardDirection = transform.forward;
        var directionToTarget = objectOfInterest.position - transform.position;

        var angle = Mathf.Acos(Vector3.Dot(forwardDirection.normalized, directionToTarget.normalized));
        
        if (angle * 100 <= _sightAngle) return true;
        
        return false;
    }

    private bool BehindObstacle(Transform objectOfInterest)
    {
        var direction = objectOfInterest.position - transform.position;
        var raycastDistance = direction.magnitude < _maxDistance ? direction.magnitude : _maxDistance;
        
        var ray = new Ray(transform.position, direction.normalized);
        var hits = new RaycastHit[16];
        var hitsAmount = Physics.RaycastNonAlloc(ray, hits, raycastDistance);
        
        Debug.DrawRay(transform.position, direction.normalized * raycastDistance);
        
        if (hitsAmount == 0) return true;
        
        foreach (var hit in hits)
        {
            if (hit.transform.TryGetComponent<Player>(out Player player) == false) // Player is empty MonoBehaviour in my case
            {
                return true;
            }

            return false;
        }

        return false;
    }
}

All the math are in InSight method. forwardDirection is the direction where an enemy is looking. When we use "targetposition" - "myposition" in directionToTarget we getting vector that pointing from our position to target. Than we use simple function to get the angle at which the player is in relation to the enemy. The function is - angle = Acos(Dot(A, B)) where A and B are normalized vectors.

In BehindObstacle we just making raycast to player position and if there are any obstacle returning true. You can set sight angle and max distance on your opinion.

And don't use this in Update method (i'm using it only for test) or very often, it can cause performance issues.