Asynchronous Tests

Apr 6, 2010 at 12:21 AM
Edited Apr 6, 2010 at 12:21 AM

I'm having hanging issues when I try to run my asynchronous tests through StatLight. If the test asserts or otherwise completes everything is fine. But StatLight seems to ignore timeouts. The StatLight process eventually exits after 5 minutes with no results shown.

Coordinator
Apr 6, 2010 at 5:13 AM

Are you running the UnitDriven async tests? Or the MSTest async tests?

If it's UnitDriven, you are probably running into the following issue.

http://statlight.codeplex.com/WorkItem/View.aspx?WorkItemId=10005

If it's because you have a single test that is "supposed" to take longer than 5 minutes. Then you probably should go vote this up http://statlight.codeplex.com/WorkItem/View.aspx?WorkItemId=10533 (as statlight expects to receive some sort of communication from the WebBrowser and after 5 minutes it currently shuts everything down). And I bet it doesn't report things nicely... (sorry haven't put much work around this scenario.

Can your tests run normally? UnitDriven - standard, or Microsoft Silverlight Testing framework standard? If so, I'm not sure what to say without gathering more details.

Hope this helps.

Apr 6, 2010 at 6:49 AM
Thanks for the quick and thorough reply. My test is an MS Test. My issue involved downloading a XAP which was then composed with MEF. Because of a malformed URI the asychronous download never completed. My timeout of 5 seconds had no effect. Perhaps I am approaching this the wrong way? I've just recently picked up the Silverlight unit testing framework so I appreciate your help.

On Apr 5, 2010, at 10:14 PM, "staxmanade" <notifications@codeplex.com> wrote:

From: staxmanade

Are you running the UnitDriven async tests? Or the MSTest async tests?

If it's UnitDriven, you are probably running into the following issue.

http://statlight.codeplex.com/WorkItem/View.aspx?WorkItemId=10005

If it's because you have a single test that is "supposed" to take longer than 5 minutes. Then you probably should go vote this up http://statlight.codeplex.com/WorkItem/View.aspx?WorkItemId=10533 (as statlight expects to receive some sort of communication from the WebBrowser and after 5 minutes it currently shuts everything down). And I bet it doesn't report things nicely... (sorry haven't put much work around this scenario.

Can your tests run normally? UnitDriven - standard, or Microsoft Silverlight Testing framework standard? If so, I'm not sure what to say without gathering more details.

Hope this helps.

Apr 6, 2010 at 4:03 PM
Edited Apr 6, 2010 at 4:03 PM

Forgot to mention that my synchronous (and asynchronous that finish for that matter) tests work fine through StatLight. Thanks for the great tool!

 

Coordinator
Apr 6, 2010 at 4:12 PM

I'm not quite sure where we are at now - were you able to get everything resolved?

If not - here's something to look at.

If you have any MEF initialization code in your App - Application_Startup - it will not be executed...

The only code executed in your test project will be code that is in a normal test class

[ClassInitialize], [TestInitialize]...etc...

hope this helps

Apr 6, 2010 at 4:49 PM
Edited Apr 6, 2010 at 4:51 PM

Thanks for bearing with me I must not be explaining things very well. Maybe some code will help. Here is the test I'm having problems with:

 

[TestMethod]
[Asynchronous]
[Timeout(5000)]
public void LoadSomeXapClassTest()
{
    var xapLoader = new XapLoader();
    xapLoader.Load_Completed += (sender, result) =>
    {
        Assert.IsNotNull(result);
        EnqueueTestComplete();
    };
    EnqueueCallback(() => xapLoader.LoadAsync(new Uri("SomeXap.xap", UriKind.RelativeOrAbsolute), typeof(SomeXapClass)));
}

Basically the XapLoader class is a wrapper around MEF which pulls in an external XAP with a DeploymentCatalog. After downloading and composing, it returns an instance of SomeXapClass which was exported. I know my wrapper class works. The only reason I'm having a problem is my test project could not resolve SomeXap.xap (my fault). But instead of throwing an exception apparently the default behavior of MEF seems to just hang forever, waiting for a callback. That shouldn't matter to me because I decorated this test with a timeout of 5 seconds. If I run this test through Visual Studio (which launches a Silverlight plugin in the browser) the timeout hits in 5 seconds and the test fails. However, if I run the test through StatLight I get the "Starting Test Run" message at which point StatLight hangs for 5 minutes then exits.

 

You can see the same behavior with this test as well:

 

[TestMethod]
[Asynchronous]
[Timeout(5000)]
public void TimeoutTest()
{
}

 

Coordinator
Apr 6, 2010 at 6:07 PM

Ahh - It's all clear now...

Looks like the Timeout property is not implemented in StatLight's version of the test runner.

In this file http://github.com/staxmanade/StatLight/blob/master/src/StatLight.Client.Harness/Hosts/MSTest/UnitTestProviders/MSTest/TestMethod.cs

here's the current code

 

public int? Timeout
{
	get { return null; }
}

Would you please...

1. Create a new Issue here on CodePlex (referencing this discussion)

2. Then here are a couple options going forward
    A. after workitem is created - wait for me to fix it - not sure when I will exactly get to it. or (shouldn't be hard to fix in the trunk, but it might not be for a week)
    B. Fork the github project - fix it, and use your version - until I can get around to it - or I pull your changes.

If you choose to fix it yourself - take a look one directory up from the TestMethod.cs file - there are a couple reflection helpers to read data from attributes.

Thanks, and hope this helps.

Apr 6, 2010 at 6:33 PM
Edited Apr 6, 2010 at 6:34 PM

Thanks for all your help. This fixes the issue for me.

TestMethod.cs

public int? Timeout
{
    get
    {
        var timeout = this.Method.GetAttribute(ProviderAttributes.TimeoutAttribute);
        return timeout == null ? null : timeout.GetObjectPropertyValue("Timeout") as int?;
    }
}

ProviderAttributes.cs

public static string TimeoutAttribute = "TimeoutAttribute";

I'm not familiar with Git so I'll let you deal with it ;-)

 

 

Apr 6, 2010 at 7:01 PM

One more thing... Though that change handles timeouts, the resulting output is not formatted for TeamCity. Not sure what that is going to take to get it to register as a "fail" but if I get some spare time I'll look into it.

 

Coordinator
Apr 8, 2010 at 4:43 PM
Edited Apr 8, 2010 at 4:44 PM

Thank you for finding this issue and your help in it's solution...

One of the side effects of adding support for the [Timeout(?)] attribute is it appears the December2008 and March2009 builds of the Microsoft Silverlight Testing Framework don't appear to work correctly with the attribute.

I decided to remove support for them going forward. It's about time to deprecate their use anyway...