Make encoding uniform before comparing strings in PHP

Question

I'm working on a feature which requires me to get the contents of a webpage, and then check to see if certain text is present in that page. It's a backlink checking tool.

The problem is this - the function runs perfectly most of the time, but occasionally, it flags a page for not having a link when the link is clearly there. I've tracked it down to the point of visually comparing the strings in the output, and they match just fine, but using the == operator, PHP tells me they don't match.

Recognizing that this is probably some sort of encoding issue, I decided to see what would happen if I used base64_encode() on them, so I could see if doing so produced different results between the two strings (which appear to be exactly the same).

My suspicions were confirmed - using base64_encode on the strings to be compared yielded a different string from each. The problem was found!

Is there some way I can make these strings uniform based on the outputted text (which matches), so that when I compare them in PHP, they match?

Answer 1

Without application code it's difficult to say what's happening.

Try using trim() on the strings to remove trailing whitespace, which is invisible to the naked eye.

You may find strcmp gives better results as well.

Answer 2

Try mb_strstr() and trim(), as pointed out by David Snabel-Caunt.

Answer 3

You could try using the DOM extension to PHP. On creating a new DOM document, you can specify the encoding of the underlying document / webpage.

According to this website, internally everything is done in UTF-8. You could then find the DOM nodes you were interested in, and compare the text content of the node

If you were not using webpages, with an associated specified character encoding, I would suggest using the multibyte functions, in particular mb_detect_encoding and mb_convert_encoding.

Answer 4

If you can't reliably get the encoding, you can use mb_convert_encoding.

$string1 = mb_convert_encoding($string1, 'utf-8', 'auto');
$string2 = mb_convert_encoding($string2, 'utf-8', 'auto');

If you can determine the encoding (from the HTTP headers or meta tags) you should specify the encoding instead of using "auto."

$string1 = mb_convert_encoding($string1, 'utf-8', $encoding1);
$string2 = mb_convert_encoding($string2, 'utf-8', $encoding2);

Answer 5

Run both through a sanitizing filter (if you have PHP greater than 5.2.0). I don't know that it will do anything, but it may.

http://www.phpro.org/tutorials/Filtering-Data-with-PHP.html#12