Lab Metrics vs. Production Metrics.

Sometimes people talk about making the Lab more like Production. Actually this is usually an anti-goal. That may sound surprising but consider this:

Production:

• lots of users
• lots of data samples
• variable available memory
• variable available disk
• variable other things running on system
• variable network quality
• variable [other stuff]
• metric goal: what is actually happening to users

In order to get the lab to look like production you’d have to start emulating those things. Now consider:

Lab:

• tiny number of test users
• small number of configs
• tightly controlled execution environment
• tightly controlled disk/memory etc.
• metric goal: have we done something bad?

The lab is designed to maximize consistency of results with an eye to detecting mistakes developers typically make. Actually getting an accurate time number is a non-goal, in fact, an anti-goal because to get accurate times/cost you would have to introduce tons of variability into the system. This would destroy the repeatability of lab metrics which is the chief goal.

If you cannot repeat a lab experiment and get the same result you have nothing!

Consequence:

• Add lab metrics to help you find mistakes (and improvements)
• Remove lab metrics that are no longer probative and replace them with new ones as needs evolve
• Never expect lab metrics to align with prod metrics (especially not distributions which are likely non-existent)
• Never count on the lab to tell you the magnitude of your mistakes/improvements, it just doesn’t have anything like the variability needed to do so, and you wouldn’t want it to
• Lab metrics are directional, not absolute

See also:

• How to Performance Lab
• Controlling Skewed Populations