Using Custom Unit Test Providers

Jan 27, 2011 at 3:52 PM

I'm using a custom unit test provider. You can see the source here:

https://github.com/bennage/Raven-Custom-Silverlight-Unit-Test-Provider

It's essentially the VSTT provider but with some wrapper classes to smooth out the async syntax. I talk about the enhancements on my blog.

After a somewhat cursory examination, it appears to me that I cannot run my tests without making modifications to statlight's source due to my dependency on a custom provider.

Are there any other options?

thanks,

Christopher

Coordinator
Jan 27, 2011 at 8:34 PM

Ok, so whatever we decide, yes changes will have to happen with StatLight to get yours to run. I think it's what changes to we make that should be discussed?

The providers in StatLight are meant to be generic enough that they just work without me having to maintain separate versions each time a new testing framework comes out. To accomplish this the providers StatLight uses rely pretty heavily on reflection based string matching for attributes. I read your original post about the async tests, but I didn't put any thought about how we could accomplish this in a generic way.

One possibility would be to change StatLight to give some sort of MEF extensibility. Calling StatLight with --OverrideTestProvider with a "RavenCustomProvider" type would trigger some sort of web service call to look for the provider xap? (that you've placed in a directory that could be streamed up to the StatLight client). It's feeling a little complicated, but I'm just throwing ideas right now.

Or perhaps instead of streaming it, it just tries to find it in the current xap. This would mean you have to manually add your assemblies into the StatLight xap and update the Manifest.

Or even better, perhaps we just use that OverrideTestProvider as a flag to not have StatLight load up a provider until all of the xap under test assemblies have loaded and we try to find the provider class inside the xap under test?

Any thoughts?

Jan 27, 2011 at 8:58 PM

I prefer looking into the current xap. Why wouldn't the dll for the provider be in the test xap?  It is for me (all reference external assemblies are). I'm new to StatLight so perhaps I'm missing something.

Overall, I like your final suggestion.

 

Hmm... I just had an idea, I'm going to fork the source and try something...

Jan 28, 2011 at 1:42 AM

Ok, well, I was trying to test out an idea where I would scan inside each test assembly for a class with a known name. Perhaps something like "StatLightUnitTestProviderProvider" (ugh, that's horrible). Then the harness could call a known method "GetProvider" and get an instance of IUnitTestProvider. That way each test assembly could be responsible for identifying it's provider.

I got stuck trying to test it (isn't that ironic) and then, well, I should do something else beside coding. I'm not sure that it was actually a useful idea anyway.

I like the suggestion you made. If I could execute 

.\StatLight.exe -x .\Raven.Tests.Silverlight.xap --OverrideTestProvider RavenCustomProvider

and have it load an instance of RavenCustomProvider from the xap itself then I would be ecstatic. I'm willing to help implement it, but I'd need a little guidance on how you want the tests written.

Coordinator
Jan 28, 2011 at 2:03 PM
Edited Jan 28, 2011 at 2:06 PM

(Disclaimer): I threw this together quickly this morning - and I know it's not working, but hoped it could help start you on your journey...

This morning (in like 5 minutes) I threw a boilerplate Silverlight project you might be able to start with (for testing against). I created a branch called CustomTestProvider and in one commit https://github.com/staxmanade/StatLight/commit/50cd2b323f89f67af03b994cc73c9f9e3931504c I have added an empty test project and a sample powershell psake task that you could use to test with.

I don't know what your experience is with psake, but I'll give a tip on how I use it. Throw the following https://github.com/staxmanade/Scripts/blob/master/Profile.common.ps1#L80 invoke-psake-locally function which I alias to "ip" into your profile. So for any dev/test your build command could be as simple as "ip build-all; ip test-custom-test-provider".

Some TODO's/thoughts of possible TODO's - unless you find a better way.

  1. When I added the project I did't add the correct reference to the Microsoft.Silverlight.Testing and VS test asssemblies. We need to reference the ones in the lib\Silverlight\Microsoft\May2010 directory.
  2. Copy a version of your Raven test provider into the /lib/Silverlight/{something} folder and reference needed assemblies in to this test project
  3. Add a number of your needed sample (async) tests to the test project
  4. Update the powershell psake test here https://github.com/staxmanade/StatLight/commit/50cd2b323f89f67af03b994cc73c9f9e3931504c#L0R765 to use the correct input.

That should be enough to get your test to work.

Steps you need to look at to make it work would be

  1. This https://github.com/staxmanade/StatLight/blob/CustomTestProvider/src/StatLight.Client.Harness.MSTest/MSTestRunnerHost.cs#L55 is where you should be able to reflect on the loaded assemblies and find the IUnitTestProvider needed.
  2. You will have to go into the StatLight console/serverside where I'm probably validating the --OverrideTestProvider input. 
    1. We can either choose to open up the --OverrideTestProvider and allow other input through
    2. Or we need to create a different command type.
    3. Or even better: just add a test provider enum type of "Custom" and in the silverlight side you can hook right into that custom one and that's where you could look for the IUnitTestProvider interface...

And I'm sure I'm missing some steps, but I hope this helps a little. I prob won't have time to work on this for a bit, but I'll try to help out where I can.

And when you're all done go here https://github.com/staxmanade/StatLight/blob/master/default.ps1#L900 and include the psake task "test-custom-test-provider" with the core test task.

Hope this helps.

Coordinator
Jan 29, 2011 at 2:43 AM

Strike about everything in the previous post.

I've pushed up all the code that "should" do the job. However it's throwing a strange exception and I can't debug at this time...

 

Jan 30, 2011 at 2:55 PM

Thanks. I started to look through the code. I suspect that multiple instances of IUnitTestProvider are being found. Since there is my custom implementation, but also the one I'm inheriting from.

What is your workflow for StatLight? I mean, how do you go about running the integration tests? I see them in the psake file, but I assume you are running just the specific one while working on the feature.

Also, when I run the build from ps I get an error about not being able to find System.Core 4.0. Do I need to hack ps to run against .NET 4 to run the build file? 

Coordinator
Jan 30, 2011 at 6:48 PM

In an earlier post I stated:

I don't know what your experience is with psake, but I'll give a tip on how I use it. Throw the following https://github.com/staxmanade/Scripts/blob/master/Profile.common.ps1#L80 invoke-psake-locally function which I alias to "ip" into your profile. So for any dev/test your build command could be as simple as "ip build-all; ip test-custom-test-provider".

That should be enough to get you in a fairly quick feedback dev cycle.

Do you have .net 4.0 framework installed? I seem to remember hacking my version of powershell at one point, but that was my previous laptop. My current laptop, I'm pretty sure I didn't hack it to run under .net 4.0 runtime... What is the specific error you are seeing? Are you sure you have the silverlight SDK 4.0 installed? Maybe my powershell build of SL isn't finding your assemblies? Not sure...

Jan 30, 2011 at 7:01 PM

:-)

Sorry, I was reading this thread in email and when you said to strike the previous post, I never went back to read it and thus missed the ps tip! Sorry.

Jan 30, 2011 at 8:15 PM

Is this the strange exception you are receiving?

System.MissingMethodException: No parameterless constructor defined for this object.
   at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& bNeedSecurityCheck)
   at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache)
   at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache)
   at System.Activator.CreateInstance(Type type, Boolean nonPublic)
   at System.Activator.CreateInstance(Type type)
   at Microsoft.Silverlight.Testing.Harness.TestClassInstanceDictionary.GetInstance(Type type)
   at Microsoft.Silverlight.Testing.Harness.UnitTestHarness.CalculateTotalMethods(AssemblyManager assemblyManager, IAssembly assembly, TestRunFilter filter)
   at Microsoft.Silverlight.Testing.Harness.UnitTestHarness.EnqueueTestAssembly(IAssembly testAssembly, TestRunFilter runFilter)
   at Microsoft.Silverlight.Testing.Harness.UnitTestHarness.PrepareTestAssemblyTasks()
   at Microsoft.Silverlight.Testing.Harness.UnitTestHarness.Initialize()
   at Microsoft.Silverlight.Testing.Harness.UnitTestHarness.Run()
   at Microsoft.Silverlight.Testing.UnitTestSystem.Run(UnitTestSettings settings)
   at Microsoft.Silverlight.Testing.UnitTestSystem.<>c__DisplayClass1.<CreateTestPage>b__0()
   at Microsoft.Silverlight.Testing.UnitTestSystem.PrepareTestService(UnitTestSettings inputSettings, Action complete)
   at Microsoft.Silverlight.Testing.UnitTestSystem.CreateTestPage(UnitTestSettings settings)
   at StatLight.Client.Harness.Hosts.MSTest.MSTestRunnerHost.StartRun()
   at StatLight.Client.Harness.Hosts.StatLightSystemBase.DisplayTestHarness()
   at StatLight.Client.Harness.Hosts.NormalStatLightSyste�

Jan 30, 2011 at 9:47 PM

I believe I found the source of the exception. I hope to have some time later tonight to commit a fix.

Coordinator
Jan 30, 2011 at 10:02 PM
Yes that was the exception I was seeing. I was stuck on what method on what object was missing...
Jan 30, 2011 at 10:27 PM

Here's the explanation for the exception.

LoadedXapData.TestAssemblies() was returning all of the assemblies in the xap, meaning that all of them were being checked for test classes. In my custom provider, I make some assumptions about tests that are a little unusual. Specifically, if a method returned IEnumerable<Task> I assumed it was a test. This was okay when using the default approach because only the one assembly was scanned. Unfortunately, one of the supporting assemblies (the async ctp one) had a method that matched, but the class containing it did not have a public parameterless constructor.

I fixed this in my provider and everything started working! I made some other tweaks and I'll have pull request for you shortly.

thanks for all the help!

Christopher

Coordinator
Jan 31, 2011 at 12:13 AM

Nice work fixing the remaining issues. I made a couple minor changes afterward, and merged your changes into master...

Nov 6, 2012 at 6:34 PM

Hi,

 

I found this discussion which is really close to my current need.

I am using AgUnit with Resharper for my Silverlight Unit Testing, so far it works great. I want to get rid of all the Enqueue methods in my testing and I found on the net the provider bennage wrote for Raven which fits my need. When I use the standard silverlight application it works fine, but it's doesn't when I launch my test from resharper.

 

This is because StatLight use his own silverlight application to run the test from resharper. The provider initialisation is done in the statup method of my silverlight application which statlight ignore. I tried to move this initilisation inside a method with the  AssemblyInitialize attribut but it didn't work, probably because it's done late ?

 

I think I need to "tweak" either StatLight.Client.For.MSTest2011December.Integration-SL5.xap or StatLight.Client.For.MSTest2011December.xap.

I modified the dll from StatLight.Client.For.MSTest2011December.Integration-SL5.xap to have the initialisation code in the app.xaml.cs but it didn't work. Now I think this is probably not the good xap, this one is maybe the one use in the TFS build process ?

This solution seems awful to me but If I try to modify the  project StatLight.Client.Harness to put inside the statup method the initialisation of the Reven Provider (and so have reference to it and to the silverlight unit testing assemblies) should it work or I am going the wrong way ? 

 

I am a bit lost on this subject so any hints on the good way to do it and I will be grateful !

Nov 6, 2012 at 8:07 PM

I found a (ugly) way to do it:

 

Adding this test in the StatLight.Core.WebServer.XapInspection.TestFileCollection.DetermineUnitTestProviderType method :

if (zipEntry.FileName.ContainsIgnoreCase("raven.tests.silverlight.unittestprovider"))
     return UnitTestProviderType.MSTestWithCustomProvider;


Other possibility (without editing the code), rename the plugin in my application to XUnitContrib.Runner.Silverlight

so it will use the MSTestWithCustomProvider.

Maybe you should add a pattern to identify all the CustomProvider and not only xunitcontrib