One of the key components of ALEX is the logic it uses to recommend insurance plans to employees. We call it The Calculator. Yes, the “T” is capitalized. Speaking the words out loud provokes thunder and dramatic music. The logic in The Calculator’s algorithm has been honed over years of analyzing data, researching use cases, and running scenarios.
So naturally, we decided to rewrite it.
Making the Leap
While a flippant decision to rewrite code for no reason wouldn’t be unprecedented for a software engineer, there was actually a lot of motivation to rewrite The Calculator. The previous version was written in Java, a fine language, but no longer a core competency at Jellyvision. In addition, our current PHP platform scales very efficiently, which is becoming more and more important as ALEX grows. Finally, the cost (both in dollars and time) of maintaining two separate hosting platforms was becoming burdensome.
We struggled with the decision to rewrite this critical piece of ALEX. There are very compelling arguments to maintain code that works well, and we tried to stick with what we had, but ultimately we were trying to fit a square peg in a round hole. Like many big decisions, once it’s made you can often tell right away whether you feel good or bad about it , and within a day we all felt good about this decision. (At least we weren’t talking about whether to rewrite it anymore.) Still, the idea of making such sweeping changes scared the crap out of many of us on the ALEX team. Our clients rely on ALEX to make accurate recommendations to their employees and they (rightly) have zero-tolerance for bad data.
With a Safety Net
How do we allay our fears? Hella data, that’s how. (Okay, it’s not quite hella data.) Over the years we have built up a lot of anonymous data describing The Calculator’s expected results in thousands and thousands of scenarios. We ran these scenarios through the old version of The Calculator to get a baseline of expected results that we can test against. Simple PHPUnit tests can then confirm that the new Calculator is functioning the same as the old version.
I for one breathe a sigh of relief watching The New Calculator running through its regression tests. One of my favorite things is when the tests fail. That’s one more defect that won’t reach our customers. The 11 minutes it takes the tests to run through all the scenarios may seem like an eternity in computer time, but given the resulting piece of mind, it’s worth every second.
An added bonus to this approach is that as we continue to tweak and improve our algorithm over time, we can visualize how our changes affect the recommendations we give to real people. We can answer questions like, “What percentage of users are affected if we weigh one metric slightly more than another?”
This is just another reminder that companies rarely regret collecting too much data or automating too many tests.
Tagged with: data • regression testing