Silverlight MSTest vNext (from Mix 2010).

Mar 30, 2010 at 2:05 PM

StatLight has been running our massive set of Silverlight Test on Teamcity for a few months now, its the 'no-pain' way to go for automating Silverlight unit tests. Great work.

 

Have you seen the new Silverlight Testing dll in the preview bits from Mix2010 (available from http://jeffatmix.com/)?

The testing framework has a new UI and a massive performance boost. Just wondering if you'd planned to pull in the new dlls to statlight to speed it up too?

Regards

Keith

Coordinator
Mar 31, 2010 at 2:41 PM
Edited Mar 31, 2010 at 2:42 PM

KeithW,

Thanks for the update. I saw parts of Jeff's talk at Mix (online), however didn't get that there was a new version to preview up on his site. I would certainly love to get this new version into StatLight.

StatLight has the ability to support multiple versions of the Microsoft Silverlight Testing Framework, unfortunately, it requires that there are no breaking changes between the different versions. (At least not breaking in the way I use the framework).

Fortunately, up until now, there hadn't been any breaking changes since the original release in Dec2008. However, there are too many breaking changes to do my usual (add assemblies to folder, update an enum, and powershell build script - rebuild StatLight and I'm done). I will have to re-architect how StatLight is built and this may take a little bit of time.

I will definitely try to get it to work. Keep an eye on the project for updates. (You can monitor the check-ins over at github.com/staxmanade/statlight)

Thanks again,

Jason

P.S. thanks for the positive feedback... (It's just enough motivation to kick me into getting the next pieces working...)

Mar 31, 2010 at 4:02 PM

Yeah I had a quick look at dropping in the dll in the Statlight codebase, but there seemed to be a fair few API changes. I didn't get far before I thought I'd check with you. I hope they haven’t reduced the functionality with the API change. Speaking of which, I'm not sure how fully baked those new dlls are, Jeff mentioned that they'd be out next April with Silverlight 4.

And no thanks for the feedback required, Statlight has saved us heaps of time and has been 'pain free' :). love it.

Keith

Coordinator
Apr 4, 2010 at 10:17 PM

I'd appreciate it if you took a look at the github March2010 branch - http://github.com/staxmanade/statlight/tree/March2010 

Check out the readme for a how to build the release version.

Let me know how it works for you.

Apr 13, 2010 at 11:26 AM

Sweet mate, will check it out ASAP.

Apr 14, 2010 at 11:49 AM

Hey again

That branch doesn't appear to be there. I downloaded the trunk and the tests fail during the build. It cant seem to dismiss the various alert boxes and Debug.Asserts automatically (i.e. the build just hangs). I built and packaged statlight by modifying default1.ps1 but our tests just dont run. Were getting a ReflectionTypeLoadExecption within StatLight.Client.Harness.Hosts.MSTest.UnitTestProviders.MSTest.UnitTestFrameworkAssembly.GetTestClasses.

But that could be a red-herring given I bypassed the tests!

Hows things going at your end, need any help?

Keith

Coordinator
Apr 14, 2010 at 3:59 PM

Sorry Keith - forgot to mention I'd merged the changes into the trunk...

I'll ramble a few of my findings so far, however won't be able to put any work into it until later...

Disclaimer: I know, I know - I should be smacked for not having the source of this C.I. project on a C.I. build itself (I haven't taken the time to figure out out how best to go about this with all the U.I. dialog assertion etc tests that have to happen) 

1. when I first downloaded the trunk from github and tried a build I got the following error

CSC : error CS0006: Metadata file '..\..\lib\Silverlight\Microsoft\March2010\Microsoft.VisualStudio.QualityTools.UnitTesting.Silverlight.dll' could not be found ".......\src\StatLight.IntegrationTests.Silverlight.OtherTestAssembly\StatLight.IntegrationTests.Silverlight.OtherTestAssembly.csproj" (Rebuild target)

(5) ->
(CoreCompile target) ->

CSC : error CS0006: Metadata file '..\..\lib\Silverlight\Microsoft\March2010\Microsoft.VisualStudio.QualityTools.UnitTesting.Silverlight.dll' could not be found

-----
I was able to reproduce this on the machine that I'm deving the project on (which was strange). After some digging around it appears that my issue is the original place I downloaded the source, unzipped, and ran build from was somehow too deep in the directory structure??? (I'll have to dig into that later) - after moving the source up a few folders I was able to get the build to work. (except on my 64 bit work computer)

3. I'm am now seeing similar issues with message boxes tests hanging on my 64 bit desktop @ work. (I'm currently deving this on a 32bit laptop @ home and haven't done a ton of testing on the 64 bit systems).

 

Would you mind creating an "Issue" around message box hangs?

If you're test projects are not hanging because of the failing message box support - you can run the following build command to get a release version (w/out running the tests)

> .\psake .\default.ps1 build-all, package-release

 

Hope this helps. 

Coordinator
Apr 15, 2010 at 6:19 AM

FYI: So I started dog-fooding StatLight tonight, and am running into some strange intermittent issues... Unfortunately I haven't tracked them down, and (although I thought everything was great - and am unsure how it couldn't be with as much testing as I have around this project) - I'm officially calling StatLight (at this current moment int time) un-stable... :(

I'm seeing issues (when using the new MS test framework) where it doesn't complete with all the tests it is supposed to have run. Sometimes generates internal errors (due to code I've changed rather recently :( ). Can't 

Give me some time to stabilize the project before putting too much effort into it.

Sorry for the inconvenience.

 

Apr 15, 2010 at 9:54 AM

Its not a problem, you've already gone out of your way to check out a beta release of the testing framework. I hope its paid off for you in terms of the performance improvement (has for us!). It’s entirely reasonable for you to park it until the next drop of the testing toolkit or whenever you have a some 'coding for fun' time :)

We can still run a unit test solution with the new Silverlight toolkit drop interactively (i.e. out of Statlight) and run the same tests via the shell (via Statlight) using the old mstest dlls. Works well.

Thanks for your efforts.

Keith

Coordinator
Apr 25, 2010 at 6:54 AM

Hey Keith,

How has your experience been with the preview of the new JeffAtMix preview of the testing framework?


If I compile the Silverlight 3.0 - April2010 silverlight toolkit and run statlight against the System.Windows.Controls.Testing.xap I observe the following

1. UI while running displays 1201 total tests (to be run)
2. StatLight reports Total 1202, Successful 1196, Failed 6 (Total time to run: 7:19)

Next I take the same source - but compile them against the new version of the Microsoft.Silverlight.Testing (Jeff at mix preview bits) and I observe the following.
1. UI while running displays 2402 total tests (to be run)
2. StatLight reports Total 1225, Successful 1202, Failed 23, (Total time to run: 5:29.97)

However each time I run against the new build I get different failure counts... And the November2009 runs are consistent.

I was curious how your thoughts on how the new build was working for you.

 

Apr 26, 2010 at 12:48 PM

Assuming you get the issues mentioned above resolved, would Silverlight 4 support be possible as well?

 

Coordinator
Apr 26, 2010 at 2:43 PM

qs7000

I'm working hard to get silverlight 4 to work. Working through the kinks now.

Keep an eye on the source commits (I'll publish once I get my full suite of tests working).

Apr 29, 2010 at 2:58 PM
Hi again We've generally had no problems with the 'Jeff at mix preview bits'. But we don’t run statlight with them, we just use it interactively for generally tdd (painful as it is). When statlight (v0.9.3631.19724) runs I believe its running with the old toolkit bits. Keith
Coordinator
Apr 30, 2010 at 1:23 AM

Hey Keith,

I've just created a v1.0 Beta build. Mind giving it a shot?

I'm still noticing some strange issues (hoping it's UI only). In the MS test runner UI - it doesn't seem to report the "total" number of tests correctly. However the results that get generated seem to be good. Let me know if you see anything strange.

Hope this works for you. (It should work with both the April 2010 and the March 2010 Microsoft.Silverlight.Testing builds.)

Thanks 

May 6, 2010 at 3:02 PM

Hello,

I tested the last build, I have a bug if a make my test project Enable running OOB.

 

 

Coordinator
May 6, 2010 at 7:11 PM

shrog,

I did a quick test (set a project to out of browser) and verified that StatLight doesn't appear to play nicely with out of browser xap. Can you create an issue around this. I will have to do some research to see how/if it's possible to support in an automated process.

Coordinator
May 8, 2010 at 5:55 PM

shrog

I've thrown up a new build that should fix your out of browser issue. Let me know if this helps.

May 8, 2010 at 10:04 PM
Edited May 8, 2010 at 10:04 PM

Well done.

Thanks