Lab Metrics vs. Production Metrics.
Sometimes people talk about making the Lab more like Production. Actually this is usually an anti-goal. That may sound surprising but consider this:
- lots of users
- lots of data samples
- variable available memory
- variable available disk
- variable other things running on system
- variable network quality
- variable [other stuff]
- metric goal: what is actually happening to users
In order to get the lab to look like production you’d have to start emulating those things. Now consider:
- tiny number of test users
- small number of configs
- tightly controlled execution environment
- tightly controlled disk/memory etc.
- metric goal: have we done something bad?
The lab is designed to maximize consistency of results with an eye to detecting mistakes developers typically make. Actually getting an accurate time number is a non-goal, in fact, an anti-goal because to get accurate times/cost you would have to introduce tons of variability into the system. This would destroy the repeatability of lab metrics which is the chief goal.
If you cannot repeat a lab experiment and get the same result you have nothing!
- Add lab metrics to help you find mistakes (and improvements)
- Remove lab metrics that are no longer probative and replace them with new ones as needs evolve
- Never expect lab metrics to align with prod metrics (especially not distributions which are likely non-existent)
- Never count on the lab to tell you the magnitude of your mistakes/improvements, it just doesn’t have anything like the variability needed to do so, and you wouldn’t want it to
- Lab metrics are directional, not absolute