'Testing in Visual Studio Succeeds Individually, Fails in a Set

When I run my tests in Visual Studio individually, they all pass without a problem. However, when I run all of them at once some pass and some fail. I tried putting in a pause of 1 second in between each test method with no success.

Any ideas? Thanks in advance for your help...

c++sharp.netvisual-studiounit-testing


Solution 1:[1]

It's very possible that some modifications/instantiations done in one test affect the others. That indicates poor test design and lack of proper isolation.

Solution 2:[2]

Everyone is probably right, some shared date is being modified between tests. But note the order of MS Test execution. Simply pausing between tests is not a solution. Each test is executed in it's own instance of the test class on a separate thread.

Solution 3:[3]

as per the other responses. It sounds like you have a singleton or a global variable that is causing the interaction.

Solution 4:[4]

Other unit test frameworks that I have used work hard to ensure that a test produces identical results whether the test is run individually or is run as part of the 'run them all' alternative. The goal is to prevent one test from having an effect on another due to side effects such as (for example) having one test leave the static state of a class in a configuration that another test is not expecting. The VS unit test framework does not appear to provide this isolation. I have 2 suggestions for minimizing the kinds of problems that the question implies. First, use non-static class in preference to a static class if the class has state (has anything other than static methods). Create a single instance of this class and have it keep the state info that was being kept in the static class. Second, if you do elect to have static class(es) with static state, have a static method that sets the static state back to 'empty' (e.g., a method that sets all static properties to null/zero/etc.). Call this at the end of each unit test to undo any effects that the test has imposed on the static state. (This is admittedly less than elegant but can be workable if done in moderation). Or do what I plan to do - find a unit test framework that provides isolation across tests.

Solution 5:[5]

I also ran into this problem although my issue ended up being a threading problem. In my case I was faking the HttpContext object since the tests relied on it's existence. However, I was setting this in the ClassInitialize method thinking this would be used for each method like below:

[ClassInitialize]
public static void ClassInit(TestContext testContext)
{
    HttpContext.Current = new HttpContext(new HttpRequest(null, "http://tempuri.org", null), new HttpResponse(null));
}

However, it turns out that each test method in the class runs in a separate thread. So I had to add this code to every test method that relied upon it to fix the issue.

[TestMethod]
public void TestMethod1()
{
    HttpContext.Current = new HttpContext(new HttpRequest(null, "http://tempuri.org", null), new HttpResponse(null));
    ...
}

[TestMethod]
public void TestMethod2()
{
    HttpContext.Current = new HttpContext(new HttpRequest(null, "http://tempuri.org", null), new HttpResponse(null));
    ...
}

See link for more information on this. http://blogs.msdn.com/b/nnaderi/archive/2007/02/17/explaining-execution-order.aspx

Solution 6:[6]

I faced a similar issue here how I solved it:

  1. Copy the code of the second test inside the first test (after it).
  2. Try to test the first test. The first test will probably fails, and then you can debug the first test (step by step) to find the static/shared variable or logic that makes the problem.

Solution 7:[7]

In my case, I had an Environment variable set with:
Environment.SetEnvironmentVariable("KEY", "value");

The implementation code for a few other tests was assuming a default value, and if the test(s) with the above line were executed first, those failed. The solution is to clean up with the following (at the end of each unit test, or in a special method for the same purpose - TestCleanup in MS/VSTest): Environment.SetEnvironmentVariable("KEY", null);
Though redundant, it is best practice to also set the environment variable to any value previously assumed by (the failing) tests to be default. Do this at the top of those unit tests (the Arrange step).

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Diadistis
Solution 2 Anthony Mastrean
Solution 3 Fortyrunner
Solution 4 Art Swri
Solution 5 james2code
Solution 6 Malick
Solution 7

Related Questions