CSLA Tests failing...

Jan 12, 2010 at 4:40 PM

I haven't tried it yet but I was forwarded an email from you to Rocky saying that most of the tests weren't working for CSLA, that some of them would succeed but then they would fail.

I was looking into the source code for StatLight trying to figure out how it worked and here is what I think might be the problem and how to fix it.

My understanding of stat light is that it will:

  1. Create a webserver in the background.
  2. Create a browser in the background.
  3. The browser loads up StatLight silverlight application.
  4. The silverlight application connects to the webserver and retrieves the xap to test via WCF.
  5. The SL app loads and exectues the xap.
  6. The SL app reports results back to the webserver via WCF.
  7. The webserver reports results back to the Console / CI server.

Does that sound correct?

So there are two potential problems with supporting UnitDriven and CSLA that hopefully we can get worked out.

  1. Many of the CSLA tests are Asynchronous, which means that they will actually finish executing the test method but that at some indeterminate time later they will report success or failure. 
    • I think what this means is that you'll have to hook into those events somehow (we can change UnitDriven if it's not possible with the current API) and only report success after either a timeout or they all tests have reported. 
    • You may be doing this already I'm not sure.
  2. Some of the tests actually connect back to the webserver and do some behavior on both the client and the server. The reason for this mostly is to test the compatibility of the serialization between .net and SL. 
    • This is arguably an integration test and not a unit test but... 
    • So I think the solution here is for StatLight to include a second flag for also allowing the test to include assemblies to load onto the server as well. In the case of CSLA the test webapp also includes WCF services that need to be available to the SL test app.
    • I think this is possible, what do you think?

If it was possible to resolve these two issues I think that would get CSLA tests to work. Additionally, It seems pretty likely that these are common scenarios for SL tests. I can create work items if you would like.

Coordinator
Jan 12, 2010 at 7:18 PM

Yes your steps are quite correct. Now were you able to figure it out because the code is readable? Or no, and you figured it out because you're just good?

When I first told Rocky that I'd try to get UnitDriven support I took a look at the CSLA testing suite (and after some time of failing miserably to get it to work (without statligh) I gave up on what StatLight would need to run things). However with you on board, we might be able to help that.

Here are some thoughts:

  1. I'm not sure how much I will be able to do inside StatLight to support the UnitDriven Async tests. I'm leveraging the MSFT Silverlight Testing Framework heavily, and I basically instruct what methods on what class to execute and then on the other side I hook into those messages the framework provides (to then ship to the web server). (Not to say give up on StatLight for UnitDriven Async tests.) 
    1. Maybe we can somehow provide a mechanism to work the Microsoft Async stuff into the UnitDriven Async???
    2. I would love if you were able to figure that out, as I'm not even sure I have a clue (at this point) how they could work together.
    3. As a start, go into StatLight and pull out the UnitDriven specific extensibility that we use to hook into the MSFT testing framework. And try to get the MSFT testing framework to work with the CSLA UnitDriven tests...
  2. With regards to (outside of StatLight) WCF services being started up by StatLight. I have an Idea.
    1. My original idea for this scenario was:
      1. Setup your build scripts to start your external web service (console app, cassini, etc) your self.
      2. Then run StatLight
      3. Kill self started web service.

        As you stated it's not really a UnitTest to support this (so why should StatLight do the work). However if, say using MEF, we could make this a peice of cake to implement... maybe it's a feature we add to StatLight. 
    2. What if StatLight defined some interface 
      public interface IExternalService : IDisposable
      {
      	// This is just an example idea/interface...
      	void Start();
      	void Stop();
      }
      
    3. We then define a "MEF" export/import for this service.
    4. We add a parameter to the console to give a directory where we tell MEF go look for instances of IExternalService

Then the StatLight application flow would look more like

  1. Create a webserver in the background.
  2. *****[MEF load and start external services]****
  3. Create a browser in the background.
  4. The browser loads up StatLight silverlight application.
  5. The silverlight application connects to the webserver and retrieves the xap to test via WCF.
  6. The SL app loads and exectues the xap.
  7. The SL app reports results back to the webserver via WCF.
  8. The webserver reports results back to the Console / CI server.
  9. *****[Shutdown MEF loaded external services]****

What do you think about this?...

One issue (have to test it to be certain - p.s. don't pre-optimization). Is what sort of overhead would this MEF lookup do each run... Now on a backend C.I.scenario, maybe not that important, but for the Desktop scenario (rapid test cycle) it is.

Jan 12, 2010 at 9:04 PM

Yes your steps are quite correct. Now were you able to figure it out because the code is readable? Or no, and you figured it out because you're just good?

I'll let you be the judge of that :) It actually took me a little snooping to figure it out but when I saw in the code that you were creating a browser object the rest sort of unfolded logically. From my experimentation to do something similar I was going down the route of creating my own SL host object via COM interop so I could execute it directly. Not only is this an incredibly painful experience I hadn't even thought about the need to have a webserver running, which complicates things of course. After thinking about it some more, given that you already solved the hard part of creating a browser and running it in a worker thread, just doing it that way seems much better overall.


Here are some thoughts:

  1. I'm not sure how much I will be able to do inside StatLight to support the UnitDriven Async tests. I'm leveraging the MSFT Silverlight Testing Framework heavily, and I basically instruct what methods on what class to execute and then on the other side I hook into those messages the framework provides (to then ship to the web server). (Not to say give up on StatLight for UnitDriven Async tests.) 
    1. Maybe we can somehow provide a mechanism to work the Microsoft Async stuff into the UnitDriven Async???
    2. I would love if you were able to figure that out, as I'm not even sure I have a clue (at this point) how they could work together.
    3. As a start, go into StatLight and pull out the UnitDriven specific extensibility that we use to hook into the MSFT testing framework. And try to get the MSFT testing framework to work with the CSLA UnitDriven tests...

I'll take a look at it. I thought I saw that you were creating a framework specific runner for each framework? I can probably figure this out if you haven't already. I'm sure it will be doable.


  1. With regards to (outside of StatLight) WCF services being started up by StatLight. I have an Idea.
    1. My original idea for this scenario was:
      1. Setup your build scripts to start your external web service (console app, cassini, etc) your self.
      2. Then run StatLight
      3. Kill self started web service.

While this would be the simpler solution I'm not totally sure this would even work to be honest... given the XSS security limitations imposed by SL. I think in order to mimick the home domain the same app that serves up the SL app has to also serve up the services (am I right about that)? Not 100% sure though.

 

  1.  
    1. What if StatLight defined some interface 
      public interface IExternalService : IDisposable
      {
      	// This is just an example idea/interface...
      	void Start();
      	void Stop();
      }
      
    2. We then define a "MEF" export/import for this service.
    3. We add a parameter to the console to give a directory where we tell MEF go look for instances of IExternalService

Actually that sounds really easy. I have worked with MEF quite a bit and I can tell you from your point of view that would be a snap.

 

Then the StatLight application flow would look more like

  1. Create a webserver in the background.
  2. *****[MEF load and start external services]****
  3. Create a browser in the background.
  4. The browser loads up StatLight silverlight application.
  5. The silverlight application connects to the webserver and retrieves the xap to test via WCF.
  6. The SL app loads and exectues the xap.
  7. The SL app reports results back to the webserver via WCF.
  8. The webserver reports results back to the Console / CI server.
  9. *****[Shutdown MEF loaded external services]****

What do you think about this?...

Sounds simple and perfect, except maybe step 9 and 8 should swap? Not a big deal.


One issue (have to test it to be certain - p.s. don't pre-optimization). Is what sort of overhead would this MEF lookup do each run... Now on a backend C.I.scenario, maybe not that important, but for the Desktop scenario (rapid test cycle) it is.

While it's not free, it's pretty darn fast, and if you're running multiple tests via MSBuild shared assemblies would only end up being loaded once also. I wouldn't worry about this too much.

 

Coordinator
Jan 18, 2010 at 11:37 PM

justinc

I spent a minute looking into the Async failing tests - (locally) have an integration test that is failing with the UnitDriven Async tests...

I created a new "Issue" around this. Where I propose one solution is to standardize on a set of messages the silverlight testing framework will send back to the server (for both MSFT and UnitDriven) scenarios. We then modify UnitDriven and hook into it to create/send these messages. And not depend on the MSFT framework to execute the UnitDriven tests.

I'm not sure an easier way to integrate the two.

I've always wanted fix up the messaging between the StatLight client/server communication anyway.

 

Now, on the server side, I did a quick spike (like 20 minutes) however bumpped in to the MEF library not being strong name signed... How do you think we go about that? Just sign it ourselves? Or hang on until it's out for good? (I'm not the best around dll signing etc... Not sure how (if we signed it ourselves) it would play with the future of statlight in .net 4.0) I'm guessing we would need different versions of statlight (3.5+ and 4.0) etc.

Thoughts?

Jan 19, 2010 at 12:35 AM

Not strong name signed? Are you using the old MEF on codeplex? If so, the good news is that it's now part of .NET proper. Just get rid of those and add a reference to System.ComponenetModel.Composability, which should be in the GAC already.

Jan 19, 2010 at 12:37 AM

But as to the message passing stuff, that sounds good. I can forsee needing to make some changes to UnitDriven to support 3rd parties tapping in to observe results however. Calling the tests to run should be easy actually, but getting the results I'm not sure how ideal it is right now.

Coordinator
Apr 25, 2010 at 6:09 AM

Hey Justin,

Any chance you could take the trunk of StatLight and run your UnitDriven stuff against it? Let me know how it goes?

May 7, 2010 at 4:12 AM
downloaded git, cloned the repo. Found out that I still have VS2010 RC on this machine. Still working on building...


On Sun, Apr 25, 2010 at 1:09 AM, staxmanade <notifications@codeplex.com> wrote:

From: staxmanade

Hey Justin,

Any chance you could take the trunk of StatLight and run your UnitDriven stuff against it? Let me know how it goes?

Read the full discussion online.

To add a post to this discussion, reply to this email (StatLight@discussions.codeplex.com)

To start a new discussion for this project, email StatLight@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe on CodePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at CodePlex.com




--
Justin Chase
http://www.justnbusiness.com