I’ve been practicing surface testing for over ten years. While I have been improving the approach, I didn’t find any major conceptual improvements to the principles already outlined.

This changed today, thanks to a colleague of mine (Arunkarthick Sundaram). He proposed the concept of system invariants. System invariants are executable assertions about data that should always be true in your system. For example: every session in your system must belong to an existing user.

My immediate objection was: we already test for invariants in surface tests. For example, if you have an invariant that says that you cannot delete a X that’s referenced by Y, you simply write a test that doesn’t allow you to delete X when it’s being referenced by Y. But doing this just with surface tests have two drawbacks: 1) you have to execute your tests perfectly; 2) the invariants are still implicit, whereas they could really become stronger by being explicit.

Invariants have a third advantage: they can be checked not just within surface tests, but also after migrations and even after certain operations that change the state of the system. The only disadvantage is that they cost some CPU cycles. But CPU cycles are cheap, and consistency issues are expensive. The extra lines are not an issue: explicit invariants are very valuable, so they don’t represent dead-weight code.

Invariants assume that you might make important omissions in both your code and your surface tests, and therefore come to your rescue. They are a verification layer that has all the advantages of surface tests: they test the system from the outside (so you can refactor the logic without touching the interface); and they test the real system without you requiring to mock anything. Invariant tests might need direct database access if they want to review data that’s not directly exposed to the user, but they can definitely be accessed from a surface if necessary. They still are close brethren to surface tests because they work at the data level of the system.

A good invariant layer, I presume, can allow you to just run certain invariants instead of all of them, for performance reasons.

My thanks to Arun for proposing something I consider a major complement to surface testing.