Are Vanity Metrics Really All That Bad?

I have a bit of a problem with all the hatred shown to so-called vanity metrics.

Eric Ries first defined vanity metrics in his landmark book, The Lean Startup. Ries says vanity metrics are the ones that most startups are judged by—things like page views, number of registered users, account activations, and things like that.

Ries says that vanity metrics are in contrast to actionable metrics. He defines an actionable metric as one that demonstrates clear cause and effect. If what causes a metric to go up or down is clear, the metric is actionable. All other metrics are vanity metrics.

I’m pretty much OK with all this so far. I’m big on action. I’ve written in my books and in posts here that if a metric will not lead to a different action, that metric is not worth gathering. I’ve said the same of estimates. If you won’t behave differently by knowing a number, don’t waste time getting that number.

So I’m fine with the definitions of “actionable” and “vanity” metrics. My problem is with some of the metrics that are thrown away as being merely vanity. For example, the number one hit on Google today when I searched for “vanity metrics” was an article on TechCrunch.

They admit to being guilty of using them and cite such metrics as 1 million downloads and 10 million registered users.

But are such numbers truly vanity metrics?

One chapter in Succeeding with Agile, is about metrics. In it, I wrote about creating a balanced scorecard and using both leading and lagging indicators. A lagging indicator is something you can measure after you have done something, and can be used to determine if you achieved a goal.

If your goal is improving quality, a lagging indicator could be the number of defects reported in the first 30 days after the release. That would tell you if you achieved your goal—but it comes with the drawback of not being at all available until 30 days after the release.

A leading indicator, on the other hand, is available in advance, and can tell you if you are on your way to achieving a goal.

The number of nightly tests that pass would be a leading indicator that a team is on its way to improving quality. The number of nightly tests passing, though, is a vanity metric in Ries’ terms. It can be easily manipulated; the team could run the same or similar tests many times to deliberately inflate the number of tests. Therefore, the linkage because cause and effect is weak. More passing tests do not guarantee improved quality.

But is the number of passing tests really a vanity metric? Is it really useless?

To show that it’s not, consider a few other metrics you’re probably familiar with: your cholesterol value, your blood pressure, your resting pulse, even your weight. A doctor can use these values and learn something about your health. If your total cholesterol value is 160, a heart attack is probably not imminent. A value of 300, though, and it’s a good thing you’re visiting your doctor.

These are leading indicators. They don’t guarantee anything. I could have a cholesterol value of 160 and have a heart attack as soon as I walk out of the doctor’s office. The only true lagging indicator would be the number of heart attacks I’ve had in the last year. Yes, absolutely a much better metric, but not available until the end of the year.

So should we avoid all vanity metrics? No. Vanity metrics can possess meaningful information. They are often leading indicators. If a website’s goal is to sell memberships then number of memberships sold is that company’s key, actionable metric.

But number of unique new visitors—a vanity metric—can be a great leading indicator. More new visitors should lead to more memberships sold. Just like more passing tests should lead to higher quality. It’s not guaranteed, but it is indicative.

The TechCrunch article I mentioned has the right attitude. It says, “Vanity metrics aren’t completely useless, just don’t be fooled by them.” The real danger of vanity metrics is that they can be gamed. We can run tests that can’t fail. We can buy traffic to our site that we know will never translate into paid memberships, but make the traffic metrics look good.

As long as no one is doing things like that, vanity metrics can serve as good leading indicators. Just keep in mind that they don’t measure what you really care about. They merely indicate whether you’re on the right path.

Last updated July 26th, 2019