Result in TFS2010 different from standalone run

Jan 5, 2011 at 11:07 AM

Hi,

I'm currently working on an integration for TFS 2010 and have a problem which I don't know how to solve.

When I build a sample testproject with two testmethods (one fails, one succeeds) and then execute StatLight in the build activity, I get different results, then when executing it directly via cmd.

As already said, the testproject has two testmethods, the succeeded one is never counted/listed. Not in Totals and not in succeeded.

I called StatLight with the same binaries and the same cmd-args and there it is working...

If you need more information just reply ask.

The solution contains two projects the .Web and the normal Silverlight Unit Test Project.

In the test project there is only one class called "Tests.cs" and its content is:

[TestClass]
    public class Tests
    {
        [TestMethod]
        public void TestMethod1()
        {
            Assert.Fail();
        }

        [TestMethod]
        public void TestMethod2()
        {
            Assert.IsNull(null);
        }
    }
=============================================

The debug output of StatLight says the following:

StatLight Execution Log: StatLight - Version 1.3.3981 Copyright (C) 2009 Jason Jarrett All Rights Reserved.  
DiskXapHostFileLoader cannot find file - C:\Users\TFS\AppData\Local\StatLight\StatLight.Client.For.December2008.xap DiskXapHostFileLoader cannot find file - C:\Users\TFS\AppData\Local\StatLight\StatLight.Client.For.March2009.xap 
****************** Input options as configured ****************** 
ContinuousIntegrationMode          : False 
ForceBrowserStart                  : False 
IsRequestingDebug                  : True 
MethodsToTest                      : IEnumerable
                                      {                                      
									  } 
MicrosoftTestingFrameworkVersion   :  
NumberOfBrowserHosts               : 1 
OutputForTeamCity                  : False 
QueryString                        :  
ShowHelp                           : False 
ShowTestingBrowserHost             : True 
StartWebServerOnly                 : False 
TagFilters                         :  
UnitTestProviderType               : Undefined 
UseRemoteTestPage                  : False 
WebBrowserType                     : SelfHosted 
XapPath                            : IEnumerable 
                                     {                                        
									  'C:\Builds\1\TestProject\StatLight Unit Testing\Binaries\testsln.slght.test.xap'
									  'C:\Builds\1\TestProject\StatLight Unit Testing\Binaries\_PublishedWebsites\testsln.slght.test.Web\ClientBin\testsln.slght.test.xap'
									 } 
XmlReportOutputPath                : C:\Builds\1\TestProject\StatLight Unit Testing\Binaries\TestResult.xml 
***************************************************************** 
RunnerType = OneTimeConsole Incoming MSTest file's hash = 357A677957F309AE85C3E5AEEDA43A32BCA23AD3 
XapItems.FilesContainedWithinXap = AppManifest.xaml 
XapItems.FilesContainedWithinXap = testsln.slght.test.dll 
XapItems.FilesContainedWithinXap = Microsoft.Silverlight.Testing.dll 
XapItems.FilesContainedWithinXap = Microsoft.VisualStudio.QualityTools.UnitTesting.Silverlight.dll 
Loading XapHost file [MSTestApril2010] 
Loading XapHost [C:\Users\TFS\AppData\Local\StatLight\StatLight.Client.For.April2010.xap] 
testPageUrlWithQueryString = http://localhost:8887/GetHtmlTestPage? 
IRunner typeof(OnetimeRunner)   
Starting Test Run: 05.01.2011 11:50:27  
Listening on http://localhost:8887/ 
Loading XapToTest C:\Builds\1\TestProject\StatLight Unit Testing\Binaries\testsln.slght.test.xap  
------------------ Test Failed ------------------ 
Test Namespace:  testsln.slght.test 
Test Class:  Tests Test Method: TestMethod1 
Exception Message:  Microsoft.VisualStudio.TestTools.UnitTesting.AssertFailedException: Assert.Fail failed.     
at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.HandleFail(String assertionName, String message, Object[] parameters)    
at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.Fail()    
at testsln.slght.test.Tests.TestMethod1() 
------------------------------------------------- 
TestExecutionMethodBeginClientEvent
   {                      
	   Started: 05.01.2011 11:50:29                   
	   MethodName: TestMethod1               
	   FullMethodName: testsln.slght.test.Tests.TestMethod1                
	   NamespaceName: testsln.slght.test                    
	   ClassName: Tests             
	   ClientEventOrder: 8       
	   ClientEventCreatedTime: 05.01.2011 11:50:29
   }   
   StatLightService.TestComplete() with 11 total messages posted - Currently have 10 registered SignalTestCompleteClientEvent
   {     
	   TotalMessagesPostedCount: 11                       
	   Failed: True            
	   TotalFailureCount: 1              
	   TotalTestsCount: 1                    
	   OtherInfo:             
	   BrowserInstanceId: 0             
	   ClientEventOrder: 10       
	   ClientEventCreatedTime: 05.01.2011 11:50:29
   }  
   Awaiting a total of 11 messages - currently have 10  
   StatLightService.TestComplete() with 10 total messages posted - Currently have 11 registered SignalTestCompleteClientEvent
   {     
	   TotalMessagesPostedCount: 10                       
	   Failed: True            
	   TotalFailureCount: 1              
	   TotalTestsCount: 1                    
	   OtherInfo:             
	   BrowserInstanceId: 0             
	   ClientEventOrder: 11       
	   ClientEventCreatedTime: 05.01.2011 11:50:29
   }  
   Awaiting a total of 11 messages - currently have 11 
   publishing TestRunCompletedServerEvent webBrowser.Stop()   
   --- 
   Completed Test Run at: 05.01.2011 11:50:29. 
   Total Run Time: 00:00:02.6520000  
   Test run results: Total 1, Successful 0, Failed 1,  
   Incoming MSTest file's hash = 357A677957F309AE85C3E5AEEDA43A32BCA23AD3 
   XapItems.FilesContainedWithinXap = AppManifest.xaml 
   XapItems.FilesContainedWithinXap = testsln.slght.test.dll 
   XapItems.FilesContainedWithinXap = Microsoft.Silverlight.Testing.dll 
   XapItems.FilesContainedWithinXap = Microsoft.VisualStudio.QualityTools.UnitTesting.Silverlight.dll 
   Loading XapHost file [MSTestApril2010] 
   Loading XapHost [C:\Users\TFS\AppData\Local\StatLight\StatLight.Client.For.April2010.xap] 
   testPageUrlWithQueryString = http://localhost:8887/GetHtmlTestPage? IRunner typeof(OnetimeRunner)   
   Starting Test Run: 05.01.2011 11:50:29  
   Listening on http://localhost:8887/ 
   Loading XapToTest C:\Builds\1\TestProject\StatLight Unit Testing\Binaries\_PublishedWebsites\testsln.slght.test.Web\ClientBin\testsln.slght.test.xap  
   ------------------ Test Failed ------------------ 
   Test Namespace:  testsln.slght.test 
   Test Class:  Tests 
   Test Method: TestMethod1 
   Exception Message:  Microsoft.VisualStudio.TestTools.UnitTesting.AssertFailedException: Assert.Fail failed.     
   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.HandleFail(String assertionName, String message, Object[] parameters)    
   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.Fail()    
   at testsln.slght.test.Tests.TestMethod1() 
   ------------------------------------------------- 
   TestExecutionMethodBeginClientEvent
   {                      
	   Started: 05.01.2011 11:50:30                   
	   MethodName: TestMethod1               
	   FullMethodName: testsln.slght.test.Tests.TestMethod1                
	   NamespaceName: testsln.slght.test                    
	   ClassName: Tests             
	   ClientEventOrder: 8       
	   ClientEventCreatedTime: 05.01.2011 11:50:31   
   }   
   StatLightService.TestComplete() with 10 total messages posted - Currently have 10 registered 
   SignalTestCompleteClientEvent
   {     
	   TotalMessagesPostedCount: 10                       
	   Failed: True            
	   TotalFailureCount: 1              
	   TotalTestsCount: 1                    
	   OtherInfo:             
	   BrowserInstanceId: 0             
	   ClientEventOrder: 10       
	   ClientEventCreatedTime: 05.01.2011 11:50:31
   }  
   Awaiting a total of 10 messages - currently have 10 publishing 
   TestRunCompletedServerEvent webBrowser.Stop()  
   StatLightService.TestComplete() with 11 total messages posted - Currently have 1 registered 
   SignalTestCompleteClientEvent
   {     
	   TotalMessagesPostedCount: 11                       
	   Failed: True            
	   TotalFailureCount: 1              
	   TotalTestsCount: 1                    
	   OtherInfo:             
	   BrowserInstanceId: 0             
	   ClientEventOrder: 1       
	   ClientEventCreatedTime: 05.01.2011 11:50:31
   }  
   Awaiting a total of 11 messages - currently have 1   
   --- 
   Completed Test Run at: 05.01.2011 11:50:31. 
   Total Run Time: 00:00:01.3260000  
   Test run results: Total 1, Successful 0, Failed 1,  
   ********************************* 
   Wrote XML report to: C:\Builds\1\TestProject\StatLight Unit Testing\Binaries\TestResult.xml 
   ********************************* 

Coordinator
Jan 6, 2011 at 4:36 AM

I'm sort-of following - but not well enough to reproduce. Any way you could zip up the project and give me the steps to reproduce?

Jan 7, 2011 at 8:07 PM

Hi, i have the same issue with CruiseControl.NET,MSBuild
It seems, when statlight executing by service - it runs only first test

namespace Scorn.SLBlog.ServiceAgent {
    public class TestClass {
        public static void TestMethod() {
            throw new NotImplementedException();
        }
    }
}

namespace Scorn.SLBlog.UnitTestsSL {
    [TestClass]
    public class Tests {
        [TestMethod]
        public void TestMethod1() {
            TestClass.TestMethod();
        }

        [TestMethod]
        public void TestMethod2() {
            TestClass.TestMethod();
        }
    }
}

 

log from msbuild - running from console:

 "C:\StatLight.v1.3\StatLight.exe" -x="../Src/Scorn.SLBlog.UnitTestsSL/bin/Debug/Scorn.SLBlog.UnitTestsSL.xap" --debug -o="MSTest" -r=".\TestsResult\TestsResultSL.xml"
  
  StatLight - Version 1.3.3981
  Copyright (C) 2009 Jason Jarrett
  All Rights Reserved.
  
  DiskXapHostFileLoader cannot find file - C:\StatLight.v1.3\StatLight.Client.For.December2008.xap
  DiskXapHostFileLoader cannot find file - C:\StatLight.v1.3\StatLight.Client.For.March2009.xap
  ****************** Input options as configured ******************
  ContinuousIntegrationMode          : False
  ForceBrowserStart                  : False
  IsRequestingDebug                  : True
  MethodsToTest                      : IEnumerable
                                       {
                                       }
  MicrosoftTestingFrameworkVersion   : 
  NumberOfBrowserHosts               : 1
  OutputForTeamCity                  : False
  QueryString                        : 
  ShowHelp                           : False
  ShowTestingBrowserHost             : False
  StartWebServerOnly                 : False
  TagFilters                         : 
  UnitTestProviderType               : MSTest
  UseRemoteTestPage                  : False
  WebBrowserType                     : SelfHosted
  XapPath                            : IEnumerable
                                       {
                                         '../Src/Scorn.SLBlog.UnitTestsSL/bin/Debug/Scorn.SLBlog.UnitTestsSL.xap'
                                       }
  XmlReportOutputPath                : .\TestsResult\TestsResultSL.xml
  *****************************************************************
  RunnerType = OneTimeConsole
  Incoming MSTest file's hash = 357A677957F309AE85C3E5AEEDA43A32BCA23AD3
  XapItems.FilesContainedWithinXap = AppManifest.xaml
  XapItems.FilesContainedWithinXap = Microsoft.Silverlight.Testing.dll
  XapItems.FilesContainedWithinXap = Microsoft.VisualStudio.QualityTools.UnitTesting.Silverlight.dll
  XapItems.FilesContainedWithinXap = Scorn.SLBlog.ServiceAgent.dll
  XapItems.FilesContainedWithinXap = Scorn.SLBlog.UnitTestsSL.dll
  Loading XapHost file [MSTestApril2010]
  Loading XapHost [C:\StatLight.v1.3\StatLight.Client.For.April2010.xap]
  testPageUrlWithQueryString = http://localhost:8887/GetHtmlTestPage?
  IRunner typeof(OnetimeRunner)
  
  
  Starting Test Run: 1/7/2011 10:43:53 PM
  
  Listening on http://localhost:8887/
  Loading XapToTest ../Src/Scorn.SLBlog.UnitTestsSL/bin/Debug/Scorn.SLBlog.UnitTestsSL.xap
  
  ------------------ Test Failed ------------------
  Test Namespace:  Scorn.SLBlog.UnitTestsSL
  Test Class:  Tests
  Test Method: TestMethod1
  Exception Message: 
  System.NotImplementedException: [Arg_NotImplementedException]
  Arguments: 
  Debugging resource strings are unavailable. Often the key and arguments provide sufficient information to diagnose the problem. See http://go.microsoft.com/fwlink/?linkid=106663&Version=4.0.51204.0&File=mscorlib.dll&Key=Arg_NotImplementedException
     at Scorn.SLBlog.UnitTestsSL.Tests.TestMethod1()
  -------------------------------------------------
  TestExecutionMethodBeginClientEvent
    {
                       Started: 1/7/2011 10:43:55 PM
                    MethodName: TestMethod1
                FullMethodName: Scorn.SLBlog.UnitTestsSL.Tests.TestMethod1
                 NamespaceName: Scorn.SLBlog.UnitTestsSL
                     ClassName: Tests
              ClientEventOrder: 7
        ClientEventCreatedTime: 1/7/2011 10:43:55 PM
    }
  
  
  ------------------ Test Failed ------------------
  Test Namespace:  Scorn.SLBlog.UnitTestsSL
  Test Class:  Tests
  Test Method: TestMethod2
  Exception Message: 
  System.NotImplementedException: [Arg_NotImplementedException]
  Arguments: 
  Debugging resource strings are unavailable. Often the key and arguments provide sufficient information to diagnose the problem. See http://go.microsoft.com/fwlink/?linkid=106663&Version=4.0.51204.0&File=mscorlib.dll&Key=Arg_NotImplementedException
     at Scorn.SLBlog.UnitTestsSL.Tests.TestMethod2()
  -------------------------------------------------
  
  StatLightService.TestComplete() with 12 total messages posted - Currently have 12 registered
  SignalTestCompleteClientEvent
    {
      TotalMessagesPostedCount: 12
                        Failed: True
             TotalFailureCount: 3
               TotalTestsCount: 3
                     OtherInfo: 
             BrowserInstanceId: 0
              ClientEventOrder: 12
        ClientEventCreatedTime: 1/7/2011 10:43:55 PM
    }
  
  Awaiting a total of 12 messages - currently have 12
  publishing TestRunCompletedServerEvent
  webBrowser.Stop()
  
  
  --- Completed Test Run at: 1/7/2011 10:43:56 PM. Total Run Time: 00:00:03.0720690
  
  Test run results: Total 2, Successful 0, Failed 2, 
  *********************************
  Wrote XML report to:
  c:\SLBlogIntegration\Build\TestsResult\TestsResultSL.xml
  *********************************
  The command ""C:\StatLight.v1.3\StatLight.exe" -x="../Src/Scorn.SLBlog.UnitTestsSL/bin/Debug/Scorn.SLBlog.UnitTestsSL.xap" --debug -o="MSTest" -r=".\TestsResult\TestsResultSL.xml"" exited with code 1.

 

log from msbuild - running by CruiseControl.NET service:

Task "Exec":
"C:\StatLight.v1.3\StatLight.exe" -x="../Src/Scorn.SLBlog.UnitTestsSL/bin/Debug/Scorn.SLBlog.UnitTestsSL.xap" --debug -o="MSTest" -r=".\TestsResult\TestsResultSL.xml"
StatLight - Version 1.3.3981
Copyright (C) 2009 Jason Jarrett
All Rights Reserved.
DiskXapHostFileLoader cannot find file - C:\StatLight.v1.3\StatLight.Client.For.December2008.xap
DiskXapHostFileLoader cannot find file - C:\StatLight.v1.3\StatLight.Client.For.March2009.xap
****************** Input options as configured ******************
ContinuousIntegrationMode : False
ForceBrowserStart : False
IsRequestingDebug : True
MethodsToTest : IEnumerable
{
}
MicrosoftTestingFrameworkVersion :
NumberOfBrowserHosts : 1
OutputForTeamCity : False
QueryString :
ShowHelp : False
ShowTestingBrowserHost : False
StartWebServerOnly : False
TagFilters :
UnitTestProviderType : MSTest
UseRemoteTestPage : False
WebBrowserType : SelfHosted
XapPath : IEnumerable
{
'../Src/Scorn.SLBlog.UnitTestsSL/bin/Debug/Scorn.SLBlog.UnitTestsSL.xap'
}
XmlReportOutputPath : .\TestsResult\TestsResultSL.xml
*****************************************************************
RunnerType = OneTimeConsole
Incoming MSTest file's hash = 357A677957F309AE85C3E5AEEDA43A32BCA23AD3
XapItems.FilesContainedWithinXap = AppManifest.xaml
XapItems.FilesContainedWithinXap = Microsoft.Silverlight.Testing.dll
XapItems.FilesContainedWithinXap = Microsoft.VisualStudio.QualityTools.UnitTesting.Silverlight.dll
XapItems.FilesContainedWithinXap = Scorn.SLBlog.ServiceAgent.dll
XapItems.FilesContainedWithinXap = Scorn.SLBlog.UnitTestsSL.dll
Loading XapHost file [MSTestApril2010]
Loading XapHost [C:\StatLight.v1.3\StatLight.Client.For.April2010.xap]
testPageUrlWithQueryString = http://localhost:8887/GetHtmlTestPage?
IRunner typeof(OnetimeRunner)
Starting Test Run: 1/7/2011 10:29:43 PM
Listening on http://localhost:8887/
Loading XapToTest ../Src/Scorn.SLBlog.UnitTestsSL/bin/Debug/Scorn.SLBlog.UnitTestsSL.xap
------------------ Test Failed ------------------
Test Namespace: Scorn.SLBlog.UnitTestsSL
Test Class: Tests
Test Method: TestMethod1
Exception Message:
System.NotImplementedException: [Arg_NotImplementedException]
Arguments:
Debugging resource strings are unavailable. Often the key and arguments provide sufficient information to diagnose the problem. See http://go.microsoft.com/fwlink/?linkid=106663&Version=4.0.51204.0&File=mscorlib.dll&Key=Arg_NotImplementedException
at Scorn.SLBlog.UnitTestsSL.Tests.TestMethod1()
-------------------------------------------------
TestExecutionMethodBeginClientEvent
{
Started: 1/7/2011 10:29:45 PM
MethodName: TestMethod1
FullMethodName: Scorn.SLBlog.UnitTestsSL.Tests.TestMethod1
NamespaceName: Scorn.SLBlog.UnitTestsSL
ClassName: Tests
ClientEventOrder: 7
ClientEventCreatedTime: 1/7/2011 10:29:45 PM
}
StatLightService.TestComplete() with 10 total messages posted - Currently have 10 registered
SignalTestCompleteClientEvent
{
TotalMessagesPostedCount: 10
Failed: True
TotalFailureCount: 1
TotalTestsCount: 1
OtherInfo:
BrowserInstanceId: 0
ClientEventOrder: 10
ClientEventCreatedTime: 1/7/2011 10:29:45 PM
}
Awaiting a total of 10 messages - currently have 10
publishing TestRunCompletedServerEvent
webBrowser.Stop()
StatLightService.TestComplete() with 11 total messages posted - Currently have 1 registered
SignalTestCompleteClientEvent
{
TotalMessagesPostedCount: 11
Failed: True
TotalFailureCount: 1
TotalTestsCount: 1
OtherInfo:
BrowserInstanceId: 0
ClientEventOrder: 1
ClientEventCreatedTime: 1/7/2011 10:29:45 PM
}
Awaiting a total of 11 messages - currently have 1
--- Completed Test Run at: 1/7/2011 10:29:45 PM. Total Run Time: 00:00:01.9998720
Test run results: Total 1, Successful 0, Failed 1,
*********************************
Wrote XML report to:
C:\SLBlogIntegration\Build\TestsResult\TestsResultSL.xml
*********************************
The command ""C:\StatLight.v1.3\StatLight.exe" -x="../Src/Scorn.SLBlog.UnitTestsSL/bin/Debug/Scorn.SLBlog.UnitTestsSL.xap" --debug -o="MSTest" -r=".\TestsResult\TestsResultSL.xml"" exited with code 1.
Done executing task "Exec" .

 

Coordinator
Jan 8, 2011 at 10:44 PM

I setup a VM and installed/got a run of statlight in CC.net. I was able to reproduce the problem. Don't have any solutions for you as of yet, but will keep digging into the problem.

Jan 9, 2011 at 12:46 AM

I can send you the TFS build activity and the corresponding .sln to reproduce if you like.

On  monday i will be in the office again. I will try to reproduce in our productive sln again. so that I can verify the issue with the probmlems mentioned by  NgastaGasta

Greetz Niels .

Coordinator
Jan 9, 2011 at 8:37 PM

NGastaGasta: I figured out your issue. You don't have the cc.net agent running with desktop interactivity. Yes it's reporting the first test, but won't report any more because desktop interactivity is not enabled. The reason it's reporting any test at all is related to a bug in the microsoft test fwk http://statlight.codeplex.com/workitem/11906

nielshebling - Are you sure all of your tfs build agents are running with desktop interactivity?

Jan 10, 2011 at 1:44 PM

staxmanade: Thank you, I set "desktop interactivity" to service and now everything works fine.

Jan 10, 2011 at 4:38 PM

Hi,

ok, when I now run my BuildService as "Interactive process" (TFS Administration Console -> Build Configuration -> BuildService -> Properties) then its working fine.

Thanks

Niels

Jan 25, 2011 at 6:40 PM

If you use TFS 2010, try the following solution : http://pyxis-tech.com/blog/2011/01/24/how-to-run-silverlight-automated-tests-on-tfs-build-server/

It explain how to combine StatLight and a TFS 2010 custom activity to break your continuous integration build when a test fail in your solution. 

P-Y