Delaying scheduled worker with RX

Question

I have an issue with delaying scheduler inside scheduled worker.

I've prepared worker class:

using System;
using System.Reactive.Concurrency;

public class Doer
{
    private readonly IDisposable schedule;
    public int Counter { get; set; } = 0;

    public Doer(IScheduler scheduler)
    {
        schedule = scheduler.Schedule(() => DoIt(() => scheduler.Sleep(TimeSpan.FromSeconds(2))));
    }

    public void Finish()
    {
        schedule.Dispose();
    }

    private void DoIt(Action delay)
    {
        while (true)
        {
            ++Counter;
            delay();
        }
    }
}

Here we have unit test:

using System;
using Microsoft.Reactive.Testing;
using Microsoft.VisualStudio.TestTools.UnitTesting;

[TestClass]
public class UnitTest
{
    [TestMethod]
    public void TestMethod1()
    {
        var scheduler = new TestScheduler();
        var doer = new Doer(scheduler);
        scheduler.AdvanceBy(TimeSpan.FromSeconds(5).Ticks);
        doer.Finish();

        Assert.AreEqual(3, doer.Counter);
    }
}

What I'm trying to get - by calling scheduler.AdvanceBy(TimeSpan.FromSeconds(5).Ticks) in test and calling scheduler.Sleep(TimeSpan.FromSeconds(2)) in worker class I would like to DoIt loop increment counter 3 times (time was advanced by 5 seconds so this loop would increment Counter 3 times in 5 seconds). Apparently that Sleep does not delay time properly. I'm sure I've missed something.