'PHP cURL doesn't consistently return 404 on a broken page

I am writing a wordpress plugin to check sites for broken links (specifically amazon links).

$ch = curl_init($single_link);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.82 Safari/537.36");
$html = curl_exec($ch);
$response_code = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
curl_close($ch);

The $response_code I receive is quite inconsistent.

At first, I received 503 errors for links that worked, at which point I implemented the user agent (obtained from my browser) and that cleared that up.

But now the links that actually are broken and should return 404 only do so some of the time. I've even pinged the same link multiple times and it seems to randomly switch between returning 404 and 503.

phpcurlphp-curl


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source

Related Questions