Chapter 5 — Measure What You Want To Improve

Google’s “long click” metric for search-result satisfaction: a user clicks a result and stays on the page a while. An unhappy user clicks, bounces back to the results, and tries again — often repeatedly. A happy user finds what they want and doesn’t return. A clever proxy for satisfaction built from behaviour either side of the click.

Use metrics to drive progress

The premise: if you can’t measure it, you can’t improve it. Too often a feature gets conceived, built, shipped and celebrated with no mechanism to tell whether it improved anything.

Good metrics do four things:

1. Focus you on the right things. They confirm your changes actually achieve your objective.
2. Guard against regressions. Visualised over time, they’re like a regression test at system scale. A dropped signup rate might surface a JS change breaking Internet Explorer; a latency spike might reveal a new feature overloading the database — both hard to spot without dashboards.
3. Drive progress forward. Code-quality and test-coverage metrics can become quality gates that fail the build below a threshold, enforcing a floor so forward progress isn’t quietly offset by low-quality work.
4. Let you compare leverage over time. If you’ve historically moved a metric ~1% per week, that becomes a baseline for future goals.

Quantifying a goal isn’t always easy. A single bug fix may not dent core metrics, but consistent bug-fixing should show up somewhere — fewer complaints, higher app-store ratings, better quality (even subjective notions can be surveyed over time). Hard to measure doesn’t mean not worth measuring. Worth asking:

• Is there a way to measure the progress of what I’m doing?
• If a task doesn’t move a core metric, is it worth doing — or is there a missing key metric?

Pick the right metric to incentivise the behaviour you want

Choosing what to measure matters as much as measuring. Engineers are good at optimising a metric once it’s set — which is exactly why the choice matters: the right metric is a North Star aligning effort; the wrong one drives ineffective or counterproductive work. Some contrasting pairs:

• Hours worked per week vs. productivity per week. Productivity and quality drop sharply once the week runs long, yet companies keep heroicising overtime — which also burns people out, killing enthusiasm and compounding the decline. Better to measure productivity via quality, site speed, user growth, user happiness.
• Click-through rate vs. long click-through rate. Similar metrics, opposite implications. Raw CTR says nothing about satisfaction; the long click uses surrounding behaviour to infer it.
• Average vs. 95th/99th-percentile response time. Optimising the average leads to very different priorities than optimising the tail.
• Bugs fixed vs. bugs outstanding. Rewarding bugs fixed quietly incentivises sloppier new code — more easy bugs to fix later.

There’s more than one way to measure any goal — and the magnitude of the target matters too. With a vague “reduce latency” goal you’ll accept small wins; with “get under 400ms” on a multi-second page you may need to cut features, re-architect, or rewrite a bottleneck in a faster language. Once the goal is aggressive, small wins stop making sense. The metric you pick shapes your decisions and behaviour.

What you don’t measure matters too. Call centres often track “average handle time” — how long to receive and “resolve” a call. Optimising it lets staff handle more calls, but pushes them to get customers off the line rather than actually solving the problem.

Instrument everything to understand what’s going on

(Section heading carried over from the original notes; no content was captured here.)

Source

Edmond Lau, The Effective Engineer, Ch. 5.