Welcome back to the new issue of “Build Your Own Testing Framework” series! As you might have noticed, currently, our testing framework only outputs failures and nothing else. It is impossible to know if it actually runs any tests when they all pass because there is no output. Today we will implement a simple reporter for our testing framework. It will report the name of the test suite and names of the tests that are being executed, for example:
This article is the fourth one of the series “Build Your Own Testing Framework”, so make sure to stick around for next parts! All articles of these series can be found here.
Shall we get started?
Render the name of the test suite
So where should the name of the test suite come from? Probably it should be a test suite class name. Currently, all of them are anonymous classes and therefore don’t have a name:
12345
runTestSuite(function(){// ^ ^// - no name here -// ...});
We would like all test suites to have that name, for example:
12345
runTestSuite(functionSpyTest(){// ^ ^// - here is the name -// ...});
We should write a test for this case:
Create a test suite with the name
Run the test suite with function runTestSuite
Assert that the test suite name is reported
Let’s try to write a test in a RunTestSuiteTest.js test suite for that:
12345
this.testItOutputsNameOfTheTest=function(){runTestSuite(functionTestSuiteName(t){});// TODO: assert that the test suite name is reported};
Now it is problematic: how are we going to assert that something is reported? Should we replace console.log(message) or process.stdout.write(message) with our own implementation, so that we can test it?:
And then we should be able to assert with: t.assertTrue(logged.indexOf("TestSuiteName") >= 0). Finally we will need to restore the old console.log function:
While this code works, it has multitude of problems:
If the test fails then oldConsoleLog function is not restored;
It has too much setup (which we could extract as a function);
It has teardown (which would be nice to avoid if we could);
It is hard to read because from 8 lines of code only 2 are delivering the core intent;
And it is testing how exactly test suite name is being reported, which is basically a View-like concern.
And fixing the last problem will actually fix everything else because this problem causes others. We can fix it by introducing some sort of Reporter type, that can respond to reportTestSuite(name) message:
12345678
this.testItOutputsNameOfTheTest=function(){runTestSuite(functionTestSuiteName(t){},{reporter:reporter});t.assertTrue(reporter.hasReportedTestSuite("TestSuiteName"));// or even better:reporter.assertHasReportedTestSuite("TestSuiteName");};
reporter in this case is some sort of test double. And what are they? - Find out here: Introducing Test Doubles.
Implementing the reporter spy
So our reporter object in the test seems terribly like a Spy Double to me, let’s test-drive it:
123456789101112131415161718192021
// test/ReporterSpyTest.jsvarrunTestSuite=require("../src/TestingFramework");varReporterSpy=require("./ReporterSpy");runTestSuite(functionReporterSpy_BehaviorTest(t){varreporter=newReporterSpy(t);// Let's write our first test:this.testAssertHasReportedTestSuite_whenFailing=function(){t.assertThrow("Expected test suite 'HelloWorld' to be reported",function(){reporter.assertHasReportedTestSuite("HelloWorld");});};});// Error: Cannot find module './ReporterSpy'// Create file test/ReporterSpy.js
Now we are getting the following error:
1234
// var reporter = new ReporterSpy(t);// ^//// TypeError: ReporterSpy is not a function
We need to create ReporterSpy object now:
123
module.exports=functionReporterSpy(assertions){};
Now we are getting:
1234
// Error: Expected to equal// Expected test suite 'HelloWorld' to be reported,// but got:// reporter.assertHasReportedTestSuite is not a function
Now we need to create a function assertHasReportedTestSuite(name) for out ReporterSpy:
123456
this.assertHasReportedTestSuite=function(expectedName){assertions.assertTrue(false,"Expected test suite 'HelloWorld' to be reported");};
Next we need to make sure, that expectedName is actually present in the error message by triangulating with different name:
12345678910111213
this.testAssertHasReportedTestSuite_whenFailing_withOtherName=function(){t.assertThrow("Expected test suite 'OtherTestSuite' to be reported",function(){reporter.assertHasReportedTestSuite("OtherTestSuite");});};// Error: Expected to equal// Expected test suite 'OtherTestSuite' to be reported,// but got:// Expected test suite 'HelloWorld' to be reported// And we need to change the respective string:"Expected test suite '"+expectedName+"' to be reported"
Then we need to make sure that we do succeed when the message is received:
this.testAssertHasReportedTestSuite_whenSucceeding=function(){t.assertNotThrow(function(){reporter.reportTestSuite("HelloWorld");reporter.assertHasReportedTestSuite("HelloWorld");});};// Error:// Expected not to throw error,// but thrown// 'reporter.reportTestSuite is not a function'// So we need to define this function in ReporterSpy:this.reportTestSuite=function(name){};// Error:// Expected not to throw error,// but thrown// 'Expected test suite 'HelloWorld' to be reported'// Now we need to provide the simplest implementation we can,// we can do that by introducing the boolean variable:module.exports=functionReporterSpy(assertions){// initially nothing is reportedvarhasReported=false;this.assertHasReportedTestSuite=function(expectedName){assertions.assertTrue(// we should fail only when nothing was reportedhasReported,"Expected test suite '"+expectedName+"' to be reported");};this.reportTestSuite=function(name){// and we mark it as reported when we do receive the messagehasReported=true;};};
And all our tests pass. Now, when the wrong name is getting reported we should still fail:
1234567891011121314151617181920212223242526272829
this.testAssertHasReportedTestSuite_whenReporting_andFailing=function(){t.assertThrow("Expected test suite 'HelloWorld' to be reported",function(){reporter.reportTestSuite("OtherTestSuite");reporter.assertHasReportedTestSuite("HelloWorld");});};// Error: Expected to throw an error,// but nothing was thrown// Now we need to actually store the name of reported test suite:module.exports=functionReporterSpy(assertions){// initially, we didn't receive any reportsvartestSuiteName=null;this.assertHasReportedTestSuite=function(expectedName){assertions.assertTrue(// we fail only if received testSuiteName is not righttestSuiteName==="HelloWorld","Expected test suite '"+expectedName+"' to be reported");};this.reportTestSuite=function(name){// and we need to store the reported nametestSuiteName=name;};};
And all tests pass again. Although, we should notice this weird condition:
1
testSuiteName==="HelloWorld"
Looks like our current production code is not generic enough, it will work well only with the expectedName equal to "HelloWorld". Let’s fix that by triangulating over this parameter:
1234567891011121314151617
this.testAssertHasReportedTestSuite_whenReporting_andFailingWithDifferentName=function(){t.assertThrow("Expected test suite 'OtherTestSuite' to be reported",function(){reporter.reportTestSuite("HelloWorld");reporter.assertHasReportedTestSuite("OtherTestSuite");});};// Error: Expected to throw an error,// but nothing was thrown// And we should fix it by actually using the `expectedName`:assertions.assertTrue(testSuiteName===expectedName,// ^ fixed here ^"Expected test suite '"+expectedName+"' to be reported");
And all the tests pass. Now we can get back to our failing test for the runTestSuite:
Implementing rendering of the name of the test suite
To implement this, first we will need to accept options parameter with sane defaults:
12345678910111213
functionrunTestSuite(testSuiteConstructor,options){options=options||{};varreporter=options.reporter||newSimpleReporter();// ...}// We have to implement this, otherwise our test suite will failfunctionSimpleReporter(){this.reportTestSuite=function(name){process.stdout.write("\n"+name+"\n");};}
After making the failing test pass and triangulating over the name of the test suite:
And all tests pass now. Unfortunately, this is the output that we see now:
1
Yeah, empty lines. This is because (function () {}).name is equal to "". We need to give proper names to all our anonymous constructors for the test suites:
123
runTestSuite(functionRunTestSuiteTest(t){...});runTestSuite(functionAssertEqualTest(t){...});// .. and so on ..
// test/ReporterSpyTest.jsthis.testAssertHasReportedTest_whenFailing=function(){t.assertThrow("Expected test 'testName' to be reported",function(){reporter.assertHasReportedTest("testName");});};// Error: Expected to equal// Expected test 'testName' to be reported,// but got:// reporter.assertHasReportedTest is not a function// We need to define assertHasReportedTest(name) method:this.assertHasReportedTest=function(expectedName){};// Error: Expected to throw an error,// but nothing was thrown// We need to make it throw the expected error:this.assertHasReportedTest=function(expectedName){assertions.assertTrue(false,"Expected test 'testName' to be reported");};// And the test passes. Message hard-codes `testName` -// we should triangulate over it:this.testAssertHasReportedTest_whenFailing_withDifferentName=function(){t.assertThrow("Expected test 'testDifferentName' to be reported",function(){reporter.assertHasReportedTest("testDifferentName");});};// Error: Expected to equal// Expected test 'testDifferentName' to be reported,// but got:// Expected test 'testName' to be reported// And to fix it:"Expected test '"+expectedName+"' to be reported"// Next test will force us to implement simple reportTest function:this.testAssertHasReportedTest_whenSucceeding=function(){t.assertNotThrow(function(){reporter.reportTest("testName");reporter.assertHasReportedTest("testName");});};// Error: reporter.reportTest is not a function// After fixing this and triangulating a bit, we get:module.exports=functionReporterSpy(assertions){vartestName=null;// ...this.assertHasReportedTest=function(expectedName){assertions.assertTrue(testName===expectedName,"Expected test '"+expectedName+"' to be reported");};this.reportTest=function(name){testName=name;};}// Finally we need ability to report multiple tests:this.testAssertHasReportedTest_whenSucceeding_withMultipleReports=function(){t.assertNotThrow(function(){reporter.reportTest("testName");reporter.reportTest("testOtherName");reporter.assertHasReportedTest("testName");});};// Error: Expected not to throw error,// but thrown 'Expected test 'testName' to be reported'// And to implement this:module.exports=functionReporterSpy(assertions){// we will store all reported names,// initially no names are reportedvartestNames=[];// ...this.assertHasReportedTest=function(expectedName){assertions.assertTrue(// check if expectedName was reportedtestNames.indexOf(expectedName)>=0,"Expected test '"+expectedName+"' to be reported");};this.reportTest=function(name){// store the reported test nametestNames.push(name);};}
Unfortunately, this does not pass our tests, because this test fails now:
123456
this.testAssertHasReportedTest_whenReporting_andFailing=function(){t.assertThrow("Expected test 'testName' to be reported",function(){reporter.reportTest("testOtherName");reporter.assertHasReportedTest("testName");});};
After an investigation, it becomes clear, that this happens because we can not re-use reporter variable defined at the higher level since all tests share the same testSuite object at the moment. We will have to move the creation of the reporter variable inside of each test:
1234567891011
this.testAssertHasReportedTest_whenReporting_andFailing=function(){varreporter=newReporterSpy(t);// ...};this.testAssertHasReportedTest_whenReporting_andFailing_withOtherName=function(){varreporter=newReporterSpy(t);// ...};// .. and so on ..
And this makes all our tests pass.
Stateless tests
This is quite a noticeable problem, that our users can be frustrated with, so we probably should make it easy on them and allow such variables to be fresh for every test. This can be achieved quite easy if we were to create a new testSuite for each test. Let’s write a simple test to show the problem:
123456789101112131415161718
// test/StatelessTest.jsvarrunTestSuite=require("../src/TestingFramework");runTestSuite(functionStatelessTest(t){varanswer=41;this.testItCanMutateVariable_andImmediatelyUseNewValue=function(){answer++;t.assertEqual(42,answer);};this.testItCanMutateVariableAgain_andGetTheSameResult=function(){answer++;t.assertEqual(42,answer);};// this fails as expected:// Error: Expected to equal 42, but got: 43});
And now let’s implement it by creating the testSuite for every test:
12345678910111213141516171819202122
functionrunTestSuite(testSuiteConstructor,options){options=options||{};varreporter=options.reporter||newSimpleReporter();reporter.reportTestSuite(testSuiteConstructor.name);vartestSuitePrototype=createTestSuite(testSuiteConstructor);// ^ we change this from `testSuite` to `testSuitePrototype` ^for(vartestNameintestSuitePrototype){if(testName.match(/^test/)){vartestSuite=createTestSuite(testSuiteConstructor);// ^ and we create our testSuite every time here ^testSuite[testName]();// ^ and run test on it ^}}}functioncreateTestSuite(testSuiteConstructor){returnnewtestSuiteConstructor(assertions);}
After doing this, we can move var reporter = new ReporterSpy(t); to the top level of the ReporterSpyTest suite again. And all the tests pass.
Implementation of the rendering of the test name
Finally, we need to make sure that the test suite, that we have written before will pass:
As expected it fails with Error: Expected test 'testSomeTestName' to be reported. After fixing it and applying triangulation once, we would end up with the following implementation:
12345678910111213141516171819
// src/TestingFramework.js in runTestSuite function:for(vartestNameintestSuitePrototype){if(testName.match(/^test/)){reporter.reportTest(testName);// ^ here is our implementation ^vartestSuite=createTestSuite(testSuiteConstructor);testSuite[testName]();}}functionSimpleReporter(){// ...// and we should not forget to implement it for real reporterthis.reportTest=function(name){process.stdout.write("\t"+name+"\n");};}
Now, it seems that both ReporterSpy and SimpleReporter are implementing the same Duck type - Reporter. What Duck Type is? - find out here: Meet Duck Type.
Contract testing all Reporter duck types
So we should test all our ducks that their public API don’t get out of sync:
The test suite name is empty. I think we need an ability to define a custom and dynamic test suite name:
Custom name for the test suite
We can achieve this by allowing any test suite to define special hook method, that will return its custom name, like testSuite.getTestSuiteName(). Let’s write a test for this:
After implementing it and triangulating over the name once the code looks like this:
123456789101112131415161718192021
functionrunTestSuite(testSuiteConstructor,options){options=options||{};varreporter=options.reporter||newSimpleReporter();vartestSuitePrototype=createTestSuite(testSuiteConstructor);reporter.reportTestSuite(getTestSuiteName(testSuiteConstructor,testSuitePrototype)// ^ this is the function that we introduced here to make it pass ^);for(vartestNameintestSuitePrototype){...}}functiongetTestSuiteName(testSuiteConstructor,testSuitePrototype){if(typeof(testSuitePrototype.getTestSuiteName)!=="function"){returntestSuiteConstructor.name;}returntestSuitePrototype.getTestSuiteName();}
Now, if we were to use this feature in our duck type tests: