Performance Pragmatism and Engineering Effort

Getting the very best results is rarely as economical as we might wish. Productivity is an important concern, and there are often trade-offs to be had. With regard to engineering effort, there’s pretty good thinking on this generally and when I taught classes in this I used a system like this one:

For every “feature” you’re building think about what letter grade it needs to get with rough definitions like these…

• A+ — the best the hardware is capable of, no compromises, “maximum effort” (you can almost never afford this)
• Fuzzy stuff in between
• D — if it was any worse you simply couldn’t use it; this is only for stuff that’s not very important and happens not very often.
• F — things have eroded to the point where the feature is literally unusable; even if it nominally “works” you can’t use it because it’s so horribly slow/fat/whatever. Nothing can get an F, because deleting the code is preferable by definition.

Now in order to make a good product you choose your letters very carefully. You pick a few A’s, probably zero or one A+. A bunch of B’s. More C’s. A few D’s where the effort isn’t worth it and you should work elsewhere because the D just doesn’t matter to customers that much. And no F’s.

If you think you need all A’s to win, you’re pretty much screwed.

If you burn your budget on B and C features, the ones that are supposed to be low effort, you’re also screwed.

The B/C/D should be quick and easy and not very impactful to the overall system — small and simple choices are very important; you want these guys to reuse framework and library assets you needed for your A features, not drag in their own stuff. But don’t waste a lot of time optimizing those algorithms, just make sure they are clean.

As defined, A+ is so hard that you’ll almost never even try for it. Maybe you’ll go for an A. Maybe go for an A+ in a codec, or a complex GPU program, something where every last percent might matter.

In short, it’s a dance. One that I can’t possibly do justice to in 600 words, but a dance. If you don’t choose to put your effort where it’s needed your competitors will eat you up because they will find smarter ways to invest. Hence don’t try to do all A+ — you’ll never ship. But don’t burn budget where you can’t afford it, because you lose that way, too.

Historically, this kind of thinking has led to very interesting choices. For instance, for maybe a decade or so, the products that make up Office (before there even was an Office) were using a p-code interpreter to run rather than using native code directly. It’s an atypical choice, but it allowed the binaries to be much smaller. On this plan, only very select areas were done in native code. The result was a richer product that was space efficient especially in lightly used features.

When you’re planning your work you’ll want to think about what result you need in an area and what result you know how to get. If your margin for mistakes is very low (i.e. if perf has to be nearly perfect) the cost will be high and the planning has to be very careful. Or as I like to quip, “the thinner the margin, the thicker the plan.”

In all of this there must be a place for pragmatism, else all is for naught.