Background subtraction using median in time

Question

I need to perform background subtraction in matlab. I have a short video of a man crossing over a wall from right to left, so it is a simple and quite stable video.

I learned about a method to estimate the background using median in time: we need to take some window of frames (let's say - 25 frames), and perform a median over time so we can get the "median image" which should be a good estimation for the BG (in that kind of video, for a large enough window, the presence of the moving object should be negligible with regards to the BG for a BG pixel). once we have the BG estimation we\ll be able to subtract it from a frame, etc.

My questions are:

1. If my video is an RGB video - how do I perform the median operation? I tried using the Hue channel of each frame, but it doesn't work (see my code below).
2. basically - what's the simplest way to do it? I'm not sure about the quality of my implementation and would like to get some advise... thanks

EDIT: the video I'm using is pretty similar to the ones that can be found here under "Classification database" --> "walk".

---

K = 31; % window size
half_win = (K+1)/2;
thresh = 0.7; % threshold for the subtraction

reader = VideoReader('input.avi');
writer  = VideoWriter('bianry.avi');
open(writer);

window = cell(1,K);
% handling first frame
[h,~,~] = rgb2hsv(im2single(readFrame(reader)));
window{half_win} = h;
k = 1;
% mirror padding of frames
while hasFrame(reader)
    if k == half_win
        break
    end
    [h,~,~] = rgb2hsv(im2single(readFrame(reader)));
    window{half_win + k} = h;
    window{half_win - k} = h;
    k = k + 1;
end
n = ndims(window{1});
bg = median(cat(n+1, window{:}), n+1);
fg = abs( window{half_win} - bg ) > thresh;
writeVideo(writer, im2uint8(fg));

while hasFrame(reader)
     [h,~,~] = rgb2hsv(im2single(readFrame(reader)));
    % update window
    window(1:K-1) = window(2:K);
    window{K} = h;
    bg = median(cat(n+1, window{:}), n+1);
    fg = abs( window{half_win} - bg ) > thresh;
    writeVideo(writer, im2uint8(fg));
end

% mirror padding for the end of the video
ind = K-1;
for k=1:(half_win - 1)
    frame_to_mirror = window{ind};
    window(1:K-1) = window(2:K);
    window{K} = frame_to_mirror;

    bg = median(cat(n+1, window{:}), n+1);
    fg = abs( window{half_win} - bg ) > thresh;
    writeVideo(writer, im2uint8(fg));
    ind = ind -2;
end

close(writer);