Are there examples of companies using differential software testing in practice?

We are looking at adding tests to our largely untested code base and the task is daunting. While the idea of TDD or similar methodologies is attractive - our main goal in the short term is to add some regression testing to find when we've unintentionally broken code. We are planning to add unit testing but, as I mentioned, it's daunting. I'm also concerned that given our lack of culture or experience of testing until now we will have a hard time maintaining the tests or ensuring that we're actually testing what matters.

One approach that seemed promising is "differential testing" as cited in

• http://ift.tt/1FNth2F or


• http://ift.tt/1NOIQRd

We would basically measure functions or endpoints that are being used what the inputs are and what the outputs are. We would record both and then test those same functions/endpoints to see that the same inputs generate the same output. We would manually filter the cases to avoid practical duplicates, or modify cases to include cases we suspect or know to cause problems, etc.

It seems that this has several advantages:

• The set-up is potentially easier - once the basic system is built, it's easy to discover many test cases. With traditional unit testing I need to manually define all the cases.


• changes in the code are potentially easier to manage. When the code changes intentionally and there is now a different output, that test will break. But I can then "accept" the new results and future tests will be compared to those results.

The main disadvantages are:

• I don't necessarily test the important variations by default - unless I manually modify or create tests


• It maybe easier to generate largely redundant tests


• It's not generally|necessarily testing units of code. It's very easy to be testing many systems together which makes the particular failure harder to pinpoint (by itself)


• because of the last two points, running the tests may be slower or more intensive because they are actually using the system.

Because the main need for testing now is regression testing, I'm less concerned about not testing units. If the code had one result before my change and now has a different results after my change, the implications are relatively simple. Either the changes are expected or unexpected. If they are unexpected then one of my recent changes affected it and its easier to pinpoint. The speed of the tests is something I'm concerned about, and I expect that unit tests could potentially be run much faster.

I'm not expecting this approach to replace proper unit testing once we get there. But I'm curious if this is an approach that people have seen or used in practice and how the results are.