This project is read-only.
1

Closed

Request: Integrated test results reported for multiple xap files

description

Currently, if I specify to run multiple xap files (-x args) in the same run of the tester application the test results are not integrated. Each xap is processed separately with results reported before the next test starts. The results of earlier tests can scroll off the screen before the last test completes.
 
I find it more useful if the last line printed is the summary of all tests run in that session of the test runner. Note that this is the behavior of other test runners such as junit.
Closed Nov 27, 2011 at 10:29 PM by staxmanade
I've re-worked the multiple xaps approach. This should be working in all scenarios now.

comments

staxmanade wrote Nov 20, 2011 at 5:15 AM

I've have attempted to give a summary at the end of your test results. (Note this does not work in the continuous mode yet). I have quite a bit of work to get that refactored...

https://github.com/staxmanade/StatLight/commit/96786056d48bbcb2fdb33bb0a62b1697ed30cdf5

You can download the latest version from a link on the home page. Let me know what you think. Thx

ChadForGood wrote Nov 22, 2011 at 4:03 PM

I see that you added a summary report of all failed tests that is printed after all tests have completed. But, I don't see a summary of pass/fail/run counts at the end. I think the big win would be that the last line printed shows the pass/fail/run counts for the sum of all tests run for all XAP files. I want to know how many tests passed/failed/ran for the set of tests I specify. And, it needs to be the last line printed since when I have several test failures, the current pass/fail/run info scrolls off the screen ... sometimes off the console buffer so I can't even scroll up to it.

My other suggestions about reporting might be considered a matter of taste, but are based on how other tools (such as junit) work and I've found value in how those tools work. FWIW, my suggestion is to only print one character for each test while it runs as kind of a console-app progress indicator. Some test runners use a dot for a successful test and an 'E' for a failed test. So, you get output like this: "...E..EE...E". After all tests have completed, then print the failed test report (like you just added). So, I'd suggest removing the test failure reporting that occurs during the test execution phase. And instead of the current (per XAP file) pass/fail/run count output, write a single final output line showing total pass/fail/run counts.

Thanks for your time and efforts.

staxmanade wrote Nov 22, 2011 at 8:54 PM

Thanks for reviewing it and giving some feedback.

Regarding the last line summary of totals. I am in complete agreement. Something I thought about at one point but spaced it during implementation. I'll take another stab.

Regarding not printing exceptions during a run. Thank you for the honest opinion. I'm still going to decline this change and at this point I have just one reason to back up my decision. Say you have a test suite with a large number of tests and it takes a long time to run through the whole suite. I like the thought of being able to review a test failure while all other tests are being run. If I had to wait 10 minutes to get the summary of results, then I might be less inclined to research the issue found early and possibly kill the run, fix before early, and re-start a new test run. To me it feels like we can possibly reduce wasted wait time. Any thoughts?

I'll try to get to the summary line soon. Again thanks for your input.

staxmanade wrote Nov 24, 2011 at 12:15 AM

I made a quick push to apply the summary of multiple xaps. Let me know how this looks to you. Thanks! One possible thing for consideration is making a single xap summary look/feel different than an aggregated version. At this point they leverage the same output method.

ChadForGood wrote Nov 24, 2011 at 1:29 AM

Look good. Nice improvement.

wrote Nov 27, 2011 at 10:29 PM

wrote Feb 14, 2013 at 9:17 PM

wrote May 16, 2013 at 10:49 AM