Issue reproducing MATLAB's imfilter function

Question

I am trying to understand how MATLAB's imfilter function works.

im = imread("cameraman.tif");

% Kernel for sharpening the image
kernel = [
     0 -1  0;
    -1  5 -1;
     0 -1  0];

im2 = zeros(size(im));

for y = 1 : size(im,1) - 3
    for x = 1 : size(im,2) - 3

        sum = 0;
        for ky = 1:3
            for kx = 1:3
                xx = x + kx - 1;
                yy = y + ky - 1;
                sum = sum + im(yy,xx)*kernel(ky,kx);
            end
        end
        
        im2(y,x) = sum;
    end
end

% Map im2 to 0 - 255
im2 = im2 - min(im2(:));
im2 = im2 / max(im2(:)) * 255;
im2 = uint8(im2);

subplot(131), imshow(im), title('Original Image')
subplot(132), imshow(imfilter(im,kernel)), title('Matlab imfilter')
subplot(133), imshow(im2), title('My filter')

Differences at the boundaries are not my concern but my result (the subplot at the right) is clearly different than what MATLAB produces (the middle subplot) although the same kernel was used.

May I know what could be the deviation? As far as I know, kernel patch will be applied to the image element-wise and be summed over the result. Could someone let me know what I am missing? Thanks.