Lab Metrics vs. Production Metrics.

Sometimes people talk about making the Lab more like Production. Actually this is usually an anti-goal. That may sound surprising but consider this:

Production:

  • lots of users

In order to get the lab to look like production you’d have to start emulating those things. Now consider:

Lab:

  • tiny number of test users

The lab is designed to maximize consistency of results with an eye to detecting mistakes developers typically make. Actually getting an accurate time number is a non-goal, in fact, an anti-goal because to get accurate times/cost you would have to introduce tons of variability into the system. This would destroy the repeatability of lab metrics which is the chief goal.

If you cannot repeat a lab experiment and get the same result you have nothing!

Consequence:

  • Add lab metrics to help you find mistakes (and improvements)

See also:

I’m a software engineer at Facebook; I specialize in software performance engineering and programming tools generally. I survived Microsoft from 1988 to 2017.