Wildcard support for specifying multiple XAPs or DLLs

Jul 4, 2011 at 12:09 PM

From what I have read in the Statlight docs , if I have multiple XAPs or DLLs , the way I can specify this on the command line is using a -x=Path to XAP 1 or -x=Path to XAP2 or a -d=Path to DLL1 -d=path to DLL2 approach. Is it possible to specify a wildcard here to specify all test DLLs or XAPs in my solution?

Coordinator
Jul 4, 2011 at 4:58 PM
Edited Jul 4, 2011 at 4:58 PM

No, you have to explicitly supply all necessary test xap/dll parameters.

The following might be a helpful start.

 

$xaps = "";
ls * -include "*test*.xap" -Recurse | %{ $xaps += '"-x=' + $_.FullName + '" ' };
StatLight.exe $xaps

 

Note the above WILL bring in both Debug and Release builds...

Hope this helps.

Jul 6, 2011 at 1:52 PM

Thanks for the reply. While specifying multiple XAPs, I observed the following

1. With the -b option set , the number of times the browser will be launched is equal to the number of XAP files specified. Is this the expected behaviour? Please confirm. If yes, then executing statlight 10 times for 10 XAPs is the same as executing statlight once with 10 -x arguments specified (with the difference being in the number of test result files maybe -> which is where I come to point 2) ?

2. On giving the same xap filename twice  i.e statlight -x="Path to XAP1" -x="Path to XAP1" .... I was expecting this time that Statlight would append the test results of the second XAP in the test results file. However what one gets to see in the test results file are just the test results from the XAP which is first executed. I was trying this with the -MSGenericTestFormat and the MSTest unit test provider.

3. Trying the normal option of giving 2 different XAP filenames, one with 4 test cases and another with 5 testcases , the behaviour observed in point 2 didnt change.. See the debug trace and the test results below for this case.


StatLight - Version 1.4.4202
Copyright (C) 2009 Jason Jarrett
All Rights Reserved.

DiskXapHostFileLoader cannot find file - C:\Users\z002sx8w\AppData\Local\Statlight\StatLight.Client.For.December2008.xap
DiskXapHostFileLoader cannot find file - C:\Users\z002sx8w\AppData\Local\Statlight\StatLight.Client.For.March2009.xap
****************** Input options as configured ******************
ContinuousIntegrationMode          : False
Dlls                               : IEnumerable<string>
                                     {
                                     }
ForceBrowserStart                  : False
IsRequestingDebug                  : True
MethodsToTest                      : IEnumerable<string>
                                     {
                                     }
MicrosoftTestingFrameworkVersion   : April2010
NumberOfBrowserHosts               : 1
OutputForTeamCity                  : False
QueryString                        :
ShowHelp                           : False
ShowTestingBrowserHost             : True
StartWebServerOnly                 : False
TagFilters                         :
TFSGenericReport                   : True
UnitTestProviderType               : MSTest
UseRemoteTestPage                  : False
WebBrowserType                     : SelfHosted
XapPaths                           : IEnumerable<string>
                                     {
                                       'D:\TFS\Prototype\src\SilverlightUnitTestFrameworkSample\SilverlightUnitTestFrameworkSample.Web\ClientBin\AnotherTest.xap'
                                       'D:\TFS\Prototype\src\SilverlightUnitTestFrameworkSample\SilverlightUnitTestFrameworkSample.Web\ClientBin\SilverlightUnitTestFrameworkSampleTest.xap'
                                     }
XmlReportOutputPath                : D:\temp\TestResult.xml
*****************************************************************
RunnerType = OneTimeConsole
Starting configuration for: D:\TFS\Prototype\src\SilverlightUnitTestFrameworkSample\SilverlightUnitTestFrameworkSample.Web\ClientBin\AnotherTest.xap
XapItems.FilesContainedWithinXap = AnotherTest.dll
XapItems.FilesContainedWithinXap = AppManifest.xaml
XapItems.FilesContainedWithinXap = Microsoft.Silverlight.Testing.dll
XapItems.FilesContainedWithinXap = Microsoft.VisualStudio.QualityTools.UnitTesting.Silverlight.dll
XapItems.FilesContainedWithinXap = SilverlightUnitTestFrameworkSample.dll
Attempting to open port at http://localhost:8887/
testPageUrlWithQueryString = http://localhost:8887/GetHtmlTestPage?
IRunner typeof(OnetimeRunner)


Starting Test Run: 06.07.2011 15:45:14

Listening on http://localhost:8887/
Loading XapHost file [MSTestApril2010]
Loading XapHost [C:\Users\z002sx8w\AppData\Local\Statlight\StatLight.Client.For.April2010.xap]
XapItems.FilesContainedWithinXap = AnotherTest.dll
XapItems.FilesContainedWithinXap = AppManifest.xaml
XapItems.FilesContainedWithinXap = Microsoft.Silverlight.Testing.dll
XapItems.FilesContainedWithinXap = Microsoft.VisualStudio.QualityTools.UnitTesting.Silverlight.dll
XapItems.FilesContainedWithinXap = SilverlightUnitTestFrameworkSample.dll
re-writing host xap with the following files
    add -  AnotherTest.dll
        updated AppManifest - AnotherTest
    -  already has file AppManifest.xaml
    -  already has file Microsoft.Silverlight.Testing.dll
    -  already has file Microsoft.VisualStudio.QualityTools.UnitTesting.Silverlight.dll
    add -  SilverlightUnitTestFrameworkSample.dll
        updated AppManifest - SilverlightUnitTestFrameworkSample
TestExecutionMethodBeginClientEvent
  {
                     Started: 06.07.2011 15:45:16
                  MethodName: Convert_Object_IsNotVisible
              FullMethodName: AnotherTest.OneMoreTest.Convert_Object_IsNotVisible
               NamespaceName: AnotherTest
                   ClassName: OneMoreTest
            ClientEventOrder: 5
      ClientEventCreatedTime: 06.07.2011 15:45:17
  }

....
StatLightService.TestComplete() with 15 total messages posted - Currently have 15 registered
SignalTestCompleteClientEvent
  {
    TotalMessagesPostedCount: 15
                      Failed: False
           TotalFailureCount: 0
             TotalTestsCount: 4
                   OtherInfo:
           BrowserInstanceId: 0
            ClientEventOrder: 15
      ClientEventCreatedTime: 06.07.2011 15:45:17
  }

Awaiting a total of 15 messages - currently have 15
publishing TestRunCompletedServerEvent
webBrowser.Stop()

StatLightService.TestComplete() with 16 total messages posted - Currently have 1 registered
SignalTestCompleteClientEvent
  {
    TotalMessagesPostedCount: 16
                      Failed: False
           TotalFailureCount: 0
             TotalTestsCount: 4
                   OtherInfo:
           BrowserInstanceId: 0
            ClientEventOrder: 1
      ClientEventCreatedTime: 06.07.2011 15:45:17
  }

Awaiting a total of 16 messages - currently have 1


--- Completed Test Run at: 06.07.2011 15:45:17. Total Run Time: 00:00:02.6860000

Test run results: Total 4, Successful 4, Failed 0,
Starting configuration for: D:\TFS\Prototype\src\SilverlightUnitTestFrameworkSample\SilverlightUnitTestFrameworkSample.Web\ClientBin\SilverlightUnitTestFrameworkSampleTest.xap
XapItems.FilesContainedWithinXap = AppManifest.xaml
XapItems.FilesContainedWithinXap = Microsoft.Silverlight.Testing.dll
XapItems.FilesContainedWithinXap = Microsoft.VisualStudio.QualityTools.UnitTesting.Silverlight.dll
XapItems.FilesContainedWithinXap = SilverlightUnitTestFrameworkSample.dll
XapItems.FilesContainedWithinXap = SilverlightUnitTestFrameworkSampleTest.dll
Attempting to open port at http://localhost:8887/
testPageUrlWithQueryString = http://localhost:8887/GetHtmlTestPage?
IRunner typeof(OnetimeRunner)


Starting Test Run: 06.07.2011 15:45:17

Listening on http://localhost:8887/
Loading XapHost file [MSTestApril2010]
Loading XapHost [C:\Users\z002sx8w\AppData\Local\Statlight\StatLight.Client.For.April2010.xap]
XapItems.FilesContainedWithinXap = AppManifest.xaml
XapItems.FilesContainedWithinXap = Microsoft.Silverlight.Testing.dll
XapItems.FilesContainedWithinXap = Microsoft.VisualStudio.QualityTools.UnitTesting.Silverlight.dll
XapItems.FilesContainedWithinXap = SilverlightUnitTestFrameworkSample.dll
XapItems.FilesContainedWithinXap = SilverlightUnitTestFrameworkSampleTest.dll
re-writing host xap with the following files
    -  already has file AppManifest.xaml
    -  already has file Microsoft.Silverlight.Testing.dll
    -  already has file Microsoft.VisualStudio.QualityTools.UnitTesting.Silverlight.dll
    add -  SilverlightUnitTestFrameworkSample.dll
        updated AppManifest - SilverlightUnitTestFrameworkSample
    add -  SilverlightUnitTestFrameworkSampleTest.dll
        updated AppManifest - SilverlightUnitTestFrameworkSampleTest
TestExecutionMethodBeginClientEvent
  {
                     Started: 06.07.2011 15:45:17
                  MethodName: Convert_Object_IsNotNull
              FullMethodName: SilverlightUnitTestFrameworkSampleTest.ConverterVisibilityTest.Convert_Object_IsNotNull
               NamespaceName: SilverlightUnitTestFrameworkSampleTest
                   ClassName: ConverterVisibilityTest
            ClientEventOrder: 7
      ClientEventCreatedTime: 06.07.2011 15:45:17
  }

.....
StatLightService.TestComplete() with 17 total messages posted - Currently have 17 registered
SignalTestCompleteClientEvent
  {
    TotalMessagesPostedCount: 17
                      Failed: False
           TotalFailureCount: 0
             TotalTestsCount: 5
                   OtherInfo:
           BrowserInstanceId: 0
            ClientEventOrder: 18
      ClientEventCreatedTime: 06.07.2011 15:45:18
  }

Awaiting a total of 17 messages - currently have 17
publishing TestRunCompletedServerEvent
webBrowser.Stop()


--- Completed Test Run at: 06.07.2011 15:45:18. Total Run Time: 00:00:00.7810000

Test run results: Total 5, Successful 5, Failed 0,
*********************************
Wrote XML report to:
D:\temp\TestResult.xml
*********************************

And here's the test results file

<SummaryResult>
  <TestName>StatLight Tests</TestName>
  <TestResult>Passed</TestResult>
  <InnerTests>
    <InnerTest>
      <TestName>AnotherTest.OneMoreTest.Convert_Object_IsNotVisible</TestName>
      <TestResult>Passed</TestResult>
      <ErrorMessage />
    </InnerTest>
    <InnerTest>
      <TestName>AnotherTest.OneMoreTest.Convert_Object_IsVisible</TestName>
      <TestResult>Passed</TestResult>
      <ErrorMessage />
    </InnerTest>
    <InnerTest>
      <TestName>AnotherTest.OneMoreTest.MyTestButtonClick_Click_TextBlockTextShouldChangeToSpecifiedText</TestName>
      <TestResult>Passed</TestResult>
      <ErrorMessage />
    </InnerTest>
    <InnerTest>
      <TestName>AnotherTest.OneMoreTest.Page_Initialized_ShouldHaveATextBlockWithSpecifiedText</TestName>
      <TestResult>Passed</TestResult>
      <ErrorMessage />
    </InnerTest>
  </InnerTests>
</SummaryResult>

Coordinator
Jul 6, 2011 at 4:01 PM

I will have to dig a little deeper.

Out of curiosity, why would you want to run with duplicate -x="Path to XAP1" -x="Path to XAP1" arguments?

Does it work with -x="Path to XAP1" -x="Path to XAP_222" for you?

Jul 6, 2011 at 9:34 PM

Out of curiosity, why would you want to run with duplicate -x="Path to XAP1" -x="Path to XAP1" argument

Because I was too lazy to create 1 more XAP just to test how the results would look with multiple XAPs

Does it work with -x="Path to XAP1" -x="Path to XAP_222" for you?

No...point 3 and the attached trace together with the summary results file discusses describes exactly this problem.

Jul 11, 2011 at 8:31 AM

I retested the above issue again. The problem with the test results being shown only for the 1st XAP specified occurs only when the MSGenericTestFormat has been specified. The test results in the absence of this argument look fine.

Coordinator
Jul 11, 2011 at 1:41 PM
Edited Jul 11, 2011 at 2:15 PM

Ahh, I see your issue. The below line is what is causing it to only report one result.

https://github.com/staxmanade/StatLight/blob/master/src/StatLight.Core/Reporting/Providers/TFS/TFS2010/MSGenericTestXmlReport.cs#L42

The problem we have now, is that the format doesn't really support multiple higher level groupings (xaps/dll's)

http://msdn.microsoft.com/en-us/library/ms243174(VS.100).aspx

Thoughts?

Sep 16, 2011 at 6:36 PM
Edited Sep 16, 2011 at 6:40 PM

One option to resolve the issue with --MSGenericTestFormat only showing one output report would be to set up one generic Test for each xap file in Visual Studio. If you set up your test project in this manner, you will get an XML output files for each Generic test you have set up.