Welcome back to the new issue of “Build Your Own Testing Framework” series! Today we are going to unit-test runTestSuite function of our testing framework. Currently, only its happy path is implicitly tested via every test of the system. Additionally, this function’s unhappy paths are, in fact, untestable at the moment.
This article is the third one of the series “Build Your Own Testing Framework”, so make sure to stick around for next parts! All articles of these series can be found here.
Today we are going to discuss Continuous Integration, what it is and what it isn’t. We will define best Check-Out/Check-In cycle. We will explore ways to train our Continuous Integration skills. And we will do some live coding on Kata with special restriction, that allows one to train such skills. Then we will draw a bottom line.
Thanks for watching! If you liked it, please share this article on social networks and follow me on twitter: @tdd_fellow.
If you have any questions or feedback for me, don’t hesitate to reach me out on Twitter: @tdd_fellow.
Parallel Change is the refactoring technique, that allows implementing backward-incompatible changes to an API in a safe manner. It consists of 3 steps:
Expand - add new functionality as an extension of the interface (e.g.: add new methods), instead of editing signatures of existing interfaces
Migrate - mark (or log a warning) existing interface as deprecated and give time to the clients of the interface to migrate to the new interface. This might be as simple as changing your own code-base one client of this interface at a time in case when you do not have 3rd party clients of these interfaces
Contract - once all clients have migrated remove old interfaces
This technique is tremendously useful when you have 3rd party clients for your API (open-source library, SaaS with REST API, etc). Also, it is as useful for normal application development because it allows such breaking changes to never break your test suite. It allows deploying your code in the middle of the refactoring, in fact, every few minutes (Continuous Integration + Continuous Deployment).
This article contains code examples in a pseudo-code.
Today we are going to unit-test existing functionality of our own testing framework, that we have test-driven with FizzBuzzKata in the previous part.
This needs to be done, since currently only happy paths are implicitly tested via FizzBuzzKata test suite, i.e.: when all the tests pass. The following unhappy paths are not tested at the moment:
when assertTrue fails, it throws an error,
when assertEqual fails, it throws an error,
when the test fails, it renders the error and fails the whole test suite run, i.e.: non-zero exit code.
Let’s check if that test actually is testing anything by trying to break the implementation of assertTrue:
1234567
// src/TestingFramework.jsassertTrue:function(condition,message){if(condition){// <- this condition was invertedthrownewError(...);}},
And if we run it, we get the expected error:
123456789
/usr/local/bin/node AssertTrueTest.js/path/to/project/src/TestingFramework.js:4thrownewError(message||"Expected to be true, but got false");^Error:Expectedtobetrue,butgotfalse..stacktrace..Processfinishedwithexitcode1
OK, it fails as expected, now we should undo our breaking change in the implementation and run the test suite again and it should pass:
1234567
// src/TestingFramework.jsassertTrue:function(condition,message){if(!condition){// <- the breaking change here have been undonethrownewError(...);}},
Now, let’s write a new test for the case, when assertTrue fails:
123456789
// test/AssertTrueTest.jsthis.testFailure=function(){try{t.assertTrue(false);}catch(error){t.assertEqual("Expected to be true, but got false",error.message);}};
And if we run the test suite, it passes. If I was confident in the previous test, that I know why it didn’t fail, here I feel a bit uncomfortable about writing these 5 lines of test code and never seeing them fail. So let’s break the code once again!
1234567
// src/TestingFramework.jsassertTrue:function(condition,message){if(!condition){thrownewError(message||"oops");// <- we have changed error message here}},
And if we undo our breaking change in the implementation the test should pass:
12345678
// src/TestingFramework.jsassertTrue:function(condition,message){if(!condition){thrownewError(message||"Expected to be true, but got false");// ^ we have restored correct message ^}},
And the last test for assertTrue to test the custom failure message:
123456789
// test/AssertTrueTest.jsthis.testCustomFailureMessage=function(){try{t.assertTrue(false,"it is not true!");}catch(error){t.assertEqual("it is not true!",error.message);}};
Even that I am confident enough, that this try { ... } catch (..) { ... } construction does the right thing, let’s be diligent about it and break the implementation of that functionality and see this test fail:
12345678
// src/TestingFramework.jsassertTrue:function(condition,message){if(!condition){thrownewError("Expected to be true, but got false");// ^ 'message || ' was removed here ^}},
If we run the test suite:
123456789
/usr/local/bin/node AssertTrueTest.js/path/to/project/src/TestingFramework.js:4thrownewError("Expected to be true, but got false");^Error:Expectedtobetrue,butgotfalse..stacktrace..Processfinishedwithexitcode1
It fails, but this change breaks assertEqual too so that we don’t see any meaningful error message now. We can figure out if that is an expected failure or not by inspecting the stack trace:
If we open the code at this stack trace frame, we will see:
1
t.assertEqual("it is not true!",error.message);
Great, this is exactly what we expected. Undoing the breaking change and running the test suite again:
12345678
// src/TestingFramework.jsassertTrue:function(condition,message){if(!condition){thrownewError(message||"Expected to be true, but got false");// ^ 'message || ' was inserted here again ^}},
Interestingly enough, I know how to break the code now and all the tests will pass:
First, let’s extract local variable in assertTrue:
123456789
// src/TestingFramework.jsassertTrue:function(condition,message){varerrorMessage=message||"Expected to be true, but got false";if(!condition){thrownewError(errorMessage);}},
This is rather interesting. It seems, that we will have to make sure, that message parameter actually gets passed to new Error(...). We can do that by applying the Triangulation technique focusing on message parameter:
123456789
// test/AssertTrueTest.jsthis.testCustomFailureMessage_withOtherMessage=function(){try{t.assertTrue(false,"should be true");}catch(error){t.assertEqual("should be true",error.message);}};
And if we run the test suite, we get our expected failure:
I think we have all required tests for assertTrue now. This is great! And have you spotted the duplication already?
Refactoring AssertTrueTest
The duplication that is present here is our try { ... } catch (..) { ... } construct. Let’s make it a completely duplicate code by extracting couple of local variables:
1234567891011121314151617
// first, let's extract the Action Under the Test variable:varaction=function(){t.assertTrue(false);};try{action();}catch(error){t.assertEqual("Expected to be true, but got false",error.message);}// next, let's extract the Expected Error Message:varexpectedMessage="Expected to be true, but got false";try{action();}catch(error){t.assertEqual(expectedMessage,error.message);}
If we apply the same refactoring for all tests in the AssertTrueTest suite, we will see the duplication:
Use this function in all tests and inline all the extracted local variables:
1234567891011121314151617
this.testFailure=function(){assertThrow("Expected to be true, but got false",function(){t.assertTrue(false);});};this.testCustomFailureMessage=function(){assertThrow("it is not true!",function(){t.assertTrue(false,"it is not true!");});};this.testCustomFailureMessage_withOtherMessage=function(){assertThrow("should be true",function(){t.assertTrue(false,"should be true");});};
Function assertThrow seems to be useful for any test, not just AssertTrueTest suite. Let’s move it to the assertions object. We will be doing that by using Parallel Change technique:
Create new functionality first;
Step by step migrate all calls to use new functionality instead of old one;
Once old functionality is not used, remove it.
The advantage of that method is that it consists of very small steps, that can be executed with confidence and each such small step never leaves the user in a red state (failing tests or compile/parsing errors).
Let’s see how this one can be applied here:
1. Create new functionality first - we can do it by copying the assertThrow function to assertions object:
12345678910111213
// src/TestingFramework.jsvarassertions={// ...assertThrow:function(expectedMessage,action){try{action();}catch(error){// `t` needs to be changed to `this` herethis.assertEqual(expectedMessage,error.message);}}};
2. Step by step migrate all calls to use new functionality instead of old one - we do it by calling assertThrow on t (our assertions object) in the test suite. Since we still haven’t removed the old assertThrow function, we can do that one function call at the time and the tests will always be green:
123456789101112131415161718192021222324
this.testFailure=function(){t.assertThrow("Expected to be true, but got false",function(){// ^ 't.' added here ^t.assertTrue(false);});};// run tests and they still pass.this.testCustomFailureMessage=function(){t.assertThrow("it is not true!",function(){// ^ 't.' added here ^t.assertTrue(false,"it is not true!");});};// run tests and they still pass.this.testCustomFailureMessage_withOtherMessage=function(){t.assertThrow("should be true",function(){// ^ 't.' added here ^t.assertTrue(false,"should be true");});};
3. Once old functionality is not used, remove it. - now we can remove our assertThrow function defined inside of the AssertTrueTest suite and run tests:
And they pass. Let’s see the complete AssertTrueTest suite again:
1234567891011121314151617181920212223242526
// test/AssertTrueTest.jsvarrunTestSuite=require("../src/TestingFramework");runTestSuite(function(t){this.testSuccess=function(){t.assertTrue(true);};this.testFailure=function(){t.assertThrow("Expected to be true, but got false",function(){t.assertTrue(false);});};this.testCustomFailureMessage=function(){t.assertThrow("it is not true!",function(){t.assertTrue(false,"it is not true!");});};this.testCustomFailureMessage_withOtherMessage=function(){t.assertThrow("should be true",function(){t.assertTrue(false,"should be true");});};});
The only problem here is that we are relying on the untested assertThrow assertion here. Let’s unit-test it.
Testing assertThrow(expectedMessage, action)
Let’s create a new test suite with first test when assertThrow succeeds:
The code can be broken without test failure by not using expectedMessage parameter:
12345678910
// src/TestingFramework.jsassertThrow:function(expectedMessage,action){try{action();}catch(error){this.assertEqual("an error message",error.message);// ^ here 'expectedMessage' was changed to constant ^}}
Let’s triangulate the code to make sure expectedMessage is used correctly by adding a new test:
1234567
// test/AssertThrowTest.jsthis.testSuccess_withDifferentExpectedMessage=function(){t.assertThrow("a different error message",function(){thrownewError("a different error message");});};
Next test is to make sure, that assertThrow is actually comparing actual and expected error messages correctly:
123456789
// test/AssertThrowTest.jsthis.testFailure=function(){t.assertThrow("Expected to equal an error message, but got: a different error message",function(){t.assertThrow("an error message",function(){thrownewError("a different error message");});});};
And it passes. The last test, that assertThrow needs is the case, when action() is not throwing any error. In that case assertThrow should fail:
123456789
// test/AssertThrowTest.jsthis.testFailure_whenActionDoesNotThrow=function(){t.assertThrow("Expected to throw an error, but nothing was thrown",function(){t.assertThrow("an error message",function(){// does nothing});});};
Oh, and that test is passing! We clearly don’t have that functionality yet. We have to add another test without usage of outer t.assertThrow to make sure that we get a test failure:
12345678910111213
this.testThrows_whenActionDoesNotThrow=function(){varhasThrown=false;try{t.assertThrow("an error message",function(){// does nothing});}catch(error){hasThrown=true;}t.assertTrue(hasThrown,"it should have thrown");};
And if we run our tests we get the expected failure:
We can fix that by verifying, that catch block is executed in the assertThrow function:
1234567891011121314
// src/TestingFramework.jsassertThrow:function(expectedMessage,action){varhasThrown=false;// <- we initialize hasThrown heretry{action();}catch(error){hasThrown=true;// <- and we set it to true in the catch blockthis.assertEqual(expectedMessage,error.message);}this.assertTrue(hasThrown);// <- and we check that it is true}
And now if we run the test suite, the original test that we were trying to write fails as expected:
// test/AssertThrowTest.jsvarrunTestSuite=require("../src/TestingFramework");runTestSuite(function(t){this.testSuccess=function(){t.assertThrow("an error message",function(){thrownewError("an error message");});};this.testSuccess_withDifferentExpectedMessage=function(){t.assertThrow("a different error message",function(){thrownewError("a different error message");});};this.testFailure=function(){t.assertThrow("Expected to equal an error message, but got: a different error message",function(){t.assertThrow("an error message",function(){thrownewError("a different error message");});});};this.testFailure_whenActionDoesNotThrow=function(){t.assertThrow("Expected to throw an error, but nothing was thrown",function(){t.assertThrow("an error message",function(){// does nothing});});};this.testThrows_whenActionDoesNotThrow=function(){varhasThrown=false;try{t.assertThrow("an error message",function(){// does nothing});}catch(error){hasThrown=true;}t.assertTrue(hasThrown,"it should have thrown");};});
Last one for today is assertEqual:
Testing assertEqual(expected, actual)
Let’s create a test suite with the first test, when assertEqual succeeds:
This test passes, and it can be broken without test failure by always comparing to 42:
12345678
// src/TestingFramework.jsassertEqual:function(expected,actual){this.assertTrue(42==actual,// <- here 'expected' is replaced with constant"Expected to equal "+expected+", but got: "+actual);},
Triangulation to fix that:
1234567891011
// testthis.testSuccess_whenExpectedIsDifferent=function(){t.assertEqual(29,29);};// Error: Expected to equal 29, but got: 29// fix implementation:expected==actual,// <- here 'expected' is restored// and the test passes
Next mutation that does not break any tests looks this way:
123456789
// src/TestingFramework.jsassertEqual:function(expected,actual){this.assertTrue(expected==actual,// "Expected to equal " + expected + ", but got: " + actual"oops"// <- replace error message);},
Let’s add the test to protect from this kind of mutation:
12345
this.testFailure=function(){t.assertThrow("Expected to equal 42, but got: 29",function(){t.assertEqual(42,29);});};
This fails with Error: oops, because assertThrow uses assertEqual. Stack trace shows, that the failure is happening here:
12
// src/TestingFramework.js in assertThrowthis.assertEqual(expectedMessage,error.message);
So that is the expected failure. We can fix it by always providing the message "Expected to equal 42, but got: 29":
12345678
// src/TestingFramework.js in assertionsassertEqual:function(expected,actual){this.assertTrue(expected==actual,"Expected to equal 42, but got: 29"// ^ here the exact constant is used ^);},
This needs some more triangulation:
12345678910111213
this.testFailure_withDifferentExpectedAndActual=function(){t.assertThrow("Expected to equal 94, but got: 1027",function(){t.assertEqual(94,1027);});};// Error: Expected to equal 42, but got: 29// fix:this.assertTrue(expected==actual,"Expected to equal "+expected+", but got: "+actual);
I think we are done now with testing the assertEqual function. The AssertEqualTest suite is looking like that now:
123456789101112131415161718192021222324
// test/AssertEqualTest.jsvarrunTestSuite=require("../src/TestingFramework");runTestSuite(function(t){this.testSuccess=function(){t.assertEqual(42,42);};this.testSuccess_whenExpectedIsDifferent=function(){t.assertEqual(29,29);};this.testFailure=function(){t.assertThrow("Expected to equal 42, but got: 29",function(){t.assertEqual(42,29);});};this.testFailure_withDifferentExpectedAndActual=function(){t.assertThrow("Expected to equal 94, but got: 1027",function(){t.assertEqual(94,1027);});};});
// src/TestingFramework.jsvarassertions={assertTrue:function(condition,message){varerrorMessage="Expected to be true, but got false";if(message){errorMessage=message;}if(!condition){thrownewError(errorMessage);}},assertEqual:function(expected,actual){this.assertTrue(expected==actual,"Expected to equal "+expected+", but got: "+actual);},assertThrow:function(expectedMessage,action){varhasThrown=false;try{action();}catch(error){hasThrown=true;this.assertEqual(expectedMessage,error.message);}this.assertTrue(hasThrown,"Expected to throw an error, but nothing was thrown");}};
Bottom Line
Congratulations! We have completely unit-tested our assertions assertTrue and assertEqual. This resulted in natural emergence of the new assertion - assertThrow. We have unit-tested it too!
Additionally, we have practiced usage of Mutational Testing and Triangulation Technique to detect missing test cases and derive them from the code. Also, we have slightly touched the Parallel Change refactoring technique - we will see more of that in the future in these series.
Now that we have unit-tested some basic assertions, we should unit-test our testing framework test runner: runTestSuite function. This will be covered in next series of “Build Your Own Testing Framework”. Stay tuned!
Thanks!
Thank you for reading, my dear reader. If you liked it, please share this article on social networks and follow me on twitter: @tdd_fellow.
If you have any questions or feedback for me, don’t hesitate to reach me out on Twitter: @tdd_fellow.
Today we are going to test-drive the testing framework without any external testing framework.
This will be done through test-driving a simple kata (FizzBuzzKata). For example:
every time we expect a test to fail and it doesn’t, this is a failing test for our testing framework, that we will be fixing,
every time we expect a test to pass and it doesn’t, this is another failing test for our testing framework, that we will be fixing.
For practical reasons, today we are going to use concrete programming language instead of pseudo-code - javascript. Except for small details, that we will point out, the techniques shown here are language-agnostic.
return FizzBuzz when the number is divisible by 3 and 5,
return string representation of number otherwise.
Writing your first test
How do we write our first test, when we don’t have a testing framework and we want to create one? - It seems, that we have to design how the test should like in our brand new testing framework.
I personally, would go with the xUnit-like design, since it is relatively simple. Given this, we might write our first test and it will look something like that:
This test should fail, because function fizzBuzz is not defined, but it doesn’t fail, since function testNormalNumberIsReturned is never called. In fact, the object with FizzBuzzKataTest is never being created.
Clearly, to fix it we need to define assertTrue on FizzBuzzKataTest object. Obviously, we do not want our user to define all their assertion for every test suite. This means, that we want to define it on FizzBuzzKataTest object outside of the definition of FizzBuzzKataTest.
There are two ways to go about it:
inheritance: make FizzBuzzKataTest inherit from some other object function assertTrue, or
composition: make FizzBuzzKataTest accept a special object with function assertTrue defined on it.
I would like to go with composition method since it gives us more flexibility in the long run:
1
functionFizzBuzzKataTest(t){...}
and the usage of assertTrue has to change appropriately:
If we run the test suite again, we will not get any failure anymore. But we were expecting assertTrue to fail, so let’s make it fail:
123
assertTrue:function(condition){thrownewError("Expected to be true, but was false");}
When we run the test suite, we get:
12345
/path/to/project/test/FizzBuzzKataTest.js:11thrownewError("Expected to be true, but got false");^Error:Expectedtobetrue,butgotfalse
Now, let’s customize the error message a bit:
123456789
this.testNormalNumberIsReturned=function(){t.assertTrue("1"===fizzBuzz(1),"Expected to equal "+"1"+", but got: "+fizzBuzz(1));}// ...assertTrue:function(condition,message){thrownewError(message||"Expected to be true, but got false");}
When running this, we are getting the expected error:
12345
/path/to/project/test/FizzBuzzKataTest.js:11thrownewError(message||"Expected to be true, but got false");^Error:Expectedtoequal1,butgot:undefined
This looks better now. Let’s fix the error now by implementing the simplest thing that could work:
123
functionfizzBuzz(number){return"1";}
And as we run our test suite we get:
12345
/path/to/project/test/FizzBuzzKataTest.js:13thrownewError(message||"Expected to be true, but got false");^Error:Expectedtoequal1,butgot:1
Oh, it should have passed the test. I know why it didn’t: we throw this error unconditionally, let’s add an appropriate if statement to assertTrue function:
If we run this code, it does not fail. That was our first green state - it took as awhile to get here. The reason for this is that we are not only test-driving FizzBuzzKata, additionally, we are writing a feature test for a non-existing testing framework. Now that we are green, we should think about refactoring, i.e.: making the structure of our code right.
Obviously, we should move our testing framework code outside of the test suite file. Probably, somewhere in src/TestingFramework.js. For that, we need to first parametrize FizzBuzzKataTest and extract the function to run the test suite.
Now it is time to move testing code to src/TestingFramework.js:
1234567891011121314
// src/TestingFramework.jsvarassertions={assertTrue:function(condition,message){if(!condition){thrownewError(message||"Expected to be true, but got false");}}};functionrunTestSuite(testSuiteConstructor){vartestSuite=newtestSuiteConstructor(assertions);testSuite.testNormalNumberIsReturned();}
If we run the test suite again, everything should pass. Somehow, I don’t feel comfortable now, let’s try to break the test suite and see if it will fail as expected:
123
functionfizzBuzz(number){return"2";// <-- "1" was changed to "2" here}
And run tests:
12345
/path/to/project/src/TestingFramework.js:4thrownewError(message||"Expected to be true, but got false");^Error:Expectedtoequal1,butgot:2
Yes, it still works as expected. We have just introduced a Mutation to our code, to see if it is still tested properly. Let’s undo the Mutation and see the test still pass. And it does.
If you look closely now, it should be possible to inline FizzBuzzKataTest definition as an argument of runTestSuite call:
And if we run our test suite, it still works. Just to check, that we are still good, let’s repeat our Mutation from the previous step. It should still fail as expected. And it does. Undo the mutation and the test is still passing. Great.
I think we are done with Refactoring step, for now, let’s get back to writing another failing test.
Writing the second test
123
this.testAnotherNormalNumberIsReturned=function(){t.assertTrue("2"===fizzBuzz(2),"Expected to equal "+"2"+", but got: "+fizzBuzz(2));};
If we run these tests, they do not fail. This is strange, let’s look at runTestSuite function again:
REMARK: this code is Javascript specific. Other programming languages will have their own way of iterating over the function/method list and calling a function by its name. Usually, it is some sort of reflection for compiled languages and meta-programming features for interpreted languages.
If we run tests now, we get the expected failure:
12345
/path/to/project/src/TestingFramework.js:4thrownewError(message||"Expected to be true, but got false");^Error:Expectedtoequal2,butgot:1
If we try to change return "1" to return "2", of course this test will pass, but the other will fail:
12345
/path/to/project/src/TestingFramework.js:4thrownewError(message||"Expected to be true, but got false");^Error:Expectedtoequal1,butgot:2
This is great for couple of reasons:
It validates, that our change to how test* functions are discovered is correct, and
We have to have a bit smarter implementation to pass both tests now:
And if we run the tests, they pass. Now, that we are in Green state, we should start refactoring. Have you noticed the duplication already?
12345
t.assertTrue("1"===fizzBuzz(1),"Expected to equal "+"1"+", but got: "+fizzBuzz(1));// and:t.assertTrue("2"===fizzBuzz(2),"Expected to equal "+"2"+", but got: "+fizzBuzz(2));
Extracting "1" and "2" as variable expected, and fizzBuzz(1) and fizzBuzz(2) as variable actual, makes these 2 lines identical:
1234567891011
this.testNormalNumberIsReturned=function(){varexpected="1";varactual=fizzBuzz(1);t.assertTrue(expected===actual,"Expected to equal "+expected+", but got: "+actual);};this.testAnotherNormalNumberIsReturned=function(){varexpected="2";varactual=fizzBuzz(2);t.assertTrue(expected===actual,"Expected to equal "+expected+", but got: "+actual);};
Specifically, this is identical:
1
t.assertTrue(expected===actual,"Expected to equal "+expected+", but got: "+actual);
This sounds like t.assertEqual(expected, actual) to me. So let’s extract it:
123456789101112
// src/TestingFramework.jsvarassertions={assertTrue:function(condition,message){...},assertEqual:function(expected,actual){this.assertTrue(expected===actual,"Expected to equal "+expected+", but got: "+actual);}}
Now, let’s use it and inline our expected and actual variables:
This looks much more readable. If we run tests, they still pass. If we try to break our code by using some Mutation, the tests fail as expected. Great, our refactoring was a success!
Let’s finish test-driving our fizzBuzz function.
Test-Driving Fizz Buzz Kata
First test for Fizz case:
123456789101112
// test:this.testFizzIsReturned=function(){t.assertEqual("Fizz",fizzBuzz(3));};// Error: Expected to equal Fizz, but got: 3// implementation:functionfizzBuzz(number){if(number===3)return"Fizz";returnnumber.toString();}
This is pretty stupid implementation, but it works for that one tests, so let’s write the test, that will break this implementation and force us to write real if condition:
123456789101112
// test:this.testFizzIsReturnedForDifferentNumber=function(){t.assertEqual("Fizz",fizzBuzz(6));};// Error: Expected to equal Fizz, but got: 6// implementation:functionfizzBuzz(number){if(number%3===0)return"Fizz";returnnumber.toString();}
This technique is called Triangulation:
the first test is to force us to write some if statement with a correct body,
second is to force us to make the condition right.
If we had an else clause, we would have had another test to make that part right.
OK, that looks like a right implementation for Fizz, let’s write the test for Buzz now:
123456789101112131415161718192021222324252627
// test:this.testBuzzIsReturned=function(){t.assertEqual("Buzz",fizzBuzz(5));};// Error: Expected to equal Buzz, but got: 5// stupid implementation:functionfizzBuzz(number){if(number===5)return"Buzz";if(number%3===0)return"Fizz";returnnumber.toString();}// Triangulation:this.testBuzzIsReturnedForDifferentNumber=function(){t.assertEqual("Buzz",fizzBuzz(10));};// Error: Expected to equal Buzz, but got: 10// correct implementation:functionfizzBuzz(number){if(number%5===0)return"Buzz";if(number%3===0)return"Fizz";returnnumber.toString();}
And finally let’s implement final requirement FizzBuzz:
1234567891011121314151617181920212223242526272829
// test:this.testFizzBuzzIsReturned=function(){t.assertEqual("FizzBuzz",fizzBuzz(15));};// Error: Expected to equal FizzBuzz, but got: Buzz// stupid implementation:functionfizzBuzz(number){if(number===15)return"FizzBuzz";if(number%5===0)return"Buzz";if(number%3===0)return"Fizz";returnnumber.toString();}// Triangulation:this.testFizzBuzzIsReturnedForDifferentNumber=function(){t.assertEqual("FizzBuzz",fizzBuzz(30));};// Error: Expected to equal FizzBuzz, but got: Buzz// correct implementation:functionfizzBuzz(number){if(number%15===0)return"FizzBuzz";if(number%5===0)return"Buzz";if(number%3===0)return"Fizz";returnnumber.toString();}
I think we are done with the implementation. FizzBuzzKata has an extended set of requirements, but they are out of the scope of this article. These requirements force us to introduce Strategy pattern and stop using this unmaintainable chain of if statements.
Refactoring this code to Strategy pattern is left as an exercise for the reader.
Bottom Line
Congratulations! Using FizzBuzzKata we have test-driven bare-bones testing framework to the point, that we can do Test-Driven Development for a simple Kata. And all that without having any testing framework in place.
Now, with this minimal framework in place, it should be possible to unit-test the framework itself, so that we can support more use cases. This will be covered in next series of “Build Your Own Testing Framework”. Stay tuned!
Thanks!
Thank you for reading, my dear reader. If you liked it, please share this article on social networks and follow me on twitter: @tdd_fellow.
If you have any questions or feedback for me, don’t hesitate to reach me out on Twitter: @tdd_fellow.
NOTE: this TL;DR includes a lot of compressed awesome information.
Theory of Sustainable Software Development
The paper introduces us to principles, policies, and practices, that are the results of applying of Constructivist Grounded Theory research method to the five projects at Pivotal Labs.
Principles
By Engendering Positive Attitudes Toward Disruption teams can transform a challenge of team disruption into a business opportunity:
team members rolling off from the project are replaced with new members,
new members are viewed as a source of fresh perspective and as an opportunity,
new members often questioned team’s assumptions (challenging “cargo culting”).
Policies & practices that Encourage Knowledge Sharing and Continuity mitigate the risk of significant knowledge loss for the project when the team is disrupted. Policies are:
Team Code Ownership, and Shared Team Schedule. While practices are:
Continuous Pair Programming, Overlapping Pair Rotation and Knowledge Pollination.
Sustainable development is not possible if the team is incurring the technical debt. Caring about Code Quality is a way to mitigate this risk. Policy here is:
Avoid Technical Debt, and practices are:
Test-Driven Development / Behavior-Driven Development and Continuous Refactoring.
Policies
Team Code Ownership: every developer is empowered to work on and to refactor any part of the team’s code.
Shared Schedule: every team member has agreed to a fixed schedule to achieve the benefits of Sustainable Software Development. Teams are, preferably, co-located.
Avoid Technical Debt: a pair creates well-crafted code without shortcuts and short-term fixes. Solutions are best and simplest for current present without over-engineering. When inherited a legacy code base, a team is actively paying down technical debt while delivering new features.
Removing Knowledge Silos Practices
Continuous Pair Programming: two developers are collaborating to write software together as their normal mode of software development. It enables knowledge transfer, reduces knowledge silos and improves code quality. Developers always work in pairs, unless exceptional circumstances arise.
Overlapping Pair Rotation: one developer rolls off the track of work and another developer rolls on, keeping the continuity of one developer at each rotation. It results in knowledge continuity. There are three strategies for spreading the knowledge via pair rotation when fighting emerging knowledge silos:
Optimizing for people rotation: developers try to pair with the person they “least recently paired with”;
Optimizing for personal preferences: developer pick with whom they will pair at each rotation, based on personal preferences;
Optimizing for context sharing: a developer who has not been working on the track for the longest track is rotating onto the track. The goal each day is for the developer leaving the track next to empower the developer who will remain on the track.
Knowledge Pollination:
Daily stand-ups create awareness of who is working on what.
Using backlog to communicate information and status of stories.
Osmotic communication: a developer overhears a discussion in other pair and offers their help.
The pair can interrupt another pair, product manager or designer to gain the needed information.
Calling out updates to the entire team.
Interruptions are encouraged - it makes the team more efficient as knowledge pollinates across the team.
Caretaking the Code Practices
Test-Driven Development/Behavior-Driven Development: developers use BDD (to describe interactions between the user and the system) and TDD (at unit test level). Design emerges from the creation and exploration of the test cases. Teams use a variety of TDD strategies, including:
testing the responsibilities and interactions, or
contract testing using mocks.
Continuous Refactoring: developers do refactoring as part of implementing feature stories. Developers are encouraged to improve the code’s design, make the code easier to read, understand, and increase the discoverability of a component based on its responsibility. The team prefers pre-refactoring, where the developer makes a complicated work to make the current story as simple and as easy as possible.
Live on Master: every pair merges their code to master many times a day. Developers may use branches to save “spikes” and to move work-in-progress between pairing stations when rotating.
Thanks!
This is where this TL;DR ends. If you are interested in the topic, go ahead and read the paper, it has the following:
Detailed description of the research method, data collection, and analysis;
Research context and one project case study;
Detailed description of Principles, Policies, and Practices of Sustainable Software Development; also anti-patterns for each policy and practice;
Theory evaluation and conclusions.
Great thanks to the original authors of the paper, researchers, and all participants!
Some time ago, I have discovered, that using your own custom test double classes, instead of a test framework makes my test code more readable and maintainable. Here is an example (pseudo-code):
They use a language, that is not relevant to the domain, therefore, they make the test less readable.
Additionally, they have knowledge of which exactly method PasswordRevealer#toggle() should call on passwordController. If we were to rename reveal method, all tests for PasswordRevealer would fail.
The same thing would happen if we were to extract methods/functions/objects out of the PasswordRevealer.
Of course, creating such test doubles yourself will involve some boilerplate code - this is a trade-off. Example:
Recently I had a lot of conversations with so many different programmers, with different backgrounds, contexts of work and opinions. What stroke me the most, is that majority feel that TDD is slower than simple automated testing, i.e., TestFirst is slower than TestAfter.
While digging deeper in their context of work, I could only agree with them: “True, in that context it will be about 50% slower”.
Most of the time, though, the context is somewhat looking like this:
We have some company ACME, that does some sort of Agile Software Development (probably Scrum);
The company ACME focuses only on the business parts of Agile Software Development;
The company ACME tried to introduce TDD as a development practice;
Everything took much longer to be done.
Enabling practice
Now, let us dive into the development practices of Agile Software Development: development practices are usually coming from Extreme Programming (XP). In XP there are 2 terms: EnablingPractice and ExploitativePractice.
Exploitative practice gives direct benefit to the team, e.g.: speed boost and quicker feedback from users and stakeholders.
Enabling practice is required for certain other exploitative practice(s) to work.
A good example of exploitative practice is Continuous Delivery. It requires Continuous Integration, Pair-Programming, and Testing to be in place. These 3 are enabling practices.
Removing slow practices
Additionally to allowing usage of practices, that make a team go faster and deliver at the higher quality level, enabling practices allow removal of practices, that make a team go slower. For example, Pair-Programming together with TDD allows removal of code review. On most of the teams (especially, of bigger size), this makes for an instant productivity boost.
Pair-Programming, TDD and Continuous Integration, additionally to enabling the team to do Continuous Delivery, also allows replacing feature-branch VCS flow with a trunk-based flow. This allows for smaller iterations and faster user feedback.
Pair-Programming Done Right
It is worth noting, that removal of code review and introducing of Continuous Delivery is only possible, if Pair-Programming is done right:
in no case, two beginners should be working in the pair;
beginners should work together with advanced beginners/competents and proficients/experts;
advanced beginners/competents should split their time in half between working with beginners and working with proficient/experts.
Terminology Beginner, AdvancedBeginner, Competent, Proficient and Expert is from Dreyfus Skill Acquisition Model.
That allows for a good trust and mentorship models in your team(s). It enables quick growth and knowledge sharing for every member of the team.
There is another enabling practice which speeds up the knowledge sharing, it is Pair Rotation, that should be done from 1 to 2 times per day, so that for small and middle-sized teams, the bigger the feature is, the higher chance, that everyone on the team have participated in its development, and therefore have enough knowledge about it.
Additionally, Pair Rotation allows for Code Detachment and removal of Code Silos. This in turn, together with TDD, enables Ruthless Refactoring, because you are not afraid:
to break the code, thanks to TDD,
to upset an owner of the Code Silo, because there is no owner, thanks to Pair Rotation.
Bottomline
I think I can go on like this forever, but I believe the idea should be clear. I will be following up with more articles in details on each technique in the future. Stay tuned!
Thank you for reading!
If you, my dear reader, have any thoughts, questions or arguments about the topic, feel free to reach out to me on twitter: @tdd_fellow.
If you liked my ideas, follow me on twitter, and, even better, provide me with your honest feedback, so that I can improve.
It is tedious, slow and error-prone to test correctness of program manually. This article is an introduction, how to make this process as simple as a hit of one button, very fast and precise.
In the last issue about Iterative Divide & Conquer we have found out, how it is tedious to manually check that our program works correctly.
We have concluded, that it is tedious and slow to manually check that the output of our print statements is correct. Additionally, it is really easy to make a mistake and to oversee incorrect lines of the output, especially, when we have tons of them. As well, this applies to any other form of manual testing, including testing the application as an End User (via clicking/tapping for UI applications, via executing tons of commands for CLI applications).
Mitigating human error-proneness via automation
Let’s get back to our example from the previous issue:
What if we could ask our computer to run english_integer_in_tens function with provided arguments and check that the output is exactly what we provided in a comment?
If we run that code, of course, we will get some sort of compilation error, or runtime error (depending on the programming language), because function equal is not defined yet. Let’s define it! Presumably, it should be comparing left argument with right argument and printing something useful on the screen.
1234567
functionequal(left,right){if(left!=right){print("Failure: "+left+" should be equal to "+right)}else{print("OK")}}
If we run the program after defining this function we will get:
12
OKOK
If we were to break the function english_integer_in_tens’s implementation, we might get something like:
12
OKFailure:twentyshouldbeequaltoforty-two
Making automation nicer
Having all this output every time we import our small library in any application going to the standard output would be annoying to say the least. How about separating the test automation from the library?
Let’s extract all our testing print-s into the separate file, import original file from it and replace all print-s with equal-s:
Running only this separate file will produce expected output:
12345
OKOK...OKOK
Importing or running directly our original library will not produce any output. This is much nicer.
Thank you for reading!
Today we have made our testing:
easier: hit of one button, or one command on the terminal;
faster: from minutes of manual verification to milliseconds of automated checks;
preciser: automated checks can’t make a mistake, if we have something except OK - we have an error, if everything is OK - the program is working correctly;
better feedback cycle: we can see if our change is correct in the matter of 1-2 seconds after making this change.
Next time we will build a fully-functional mini testing framework. Stay tuned!