Startups love numbers. Data is king in the A/B testing world. Numbers allow entrepreneurs to quickly quantitative decisions with hard data. In this data-driven world it’s easy to forget that data without intelligence is dangerously misleading.

For example:

Back during World War II, the RAF lost a lot of planes to German anti-aircraft fire. So they decided to armor them up. But where to put the armor? The obvious answer was to look at planes that returned from missions, count up all the bullet holes in various places, and then put extra armor in the areas that attracted the most fire.

Obvious but wrong. As Hungarian-born mathematician Abraham Wald explained at the time, if a plane makes it back safely even though it has, say, a bunch of bullet holes in its wings, it means that bullet holes in the wings aren’t very dangerous. What you really want to do is armor up the areas that, on average, don’t have any bullet holes. Why? Because planes with bullet holes in those places never made it back. That’s why you don’t see any bullet holes there on the ones that do return.

From Kevin Drum’s article on MotherJones.com

I see a lot of startups focus solely on increasing the number of users they have. I’ve received pitches from startups that say things like “Over 5,000 users have already signed up!”, and I’ll say “That’s fantastic but how many people are actively using your service?”. Total users are just bullet holes in the wing.

The right way to view total users

Looking at total users is a useful way to determine demand for an idea. If you haven’t launched yet and you already have over 1,000 users sign up, you can start gauging the market. Total users can help you quantify a market but it’s a poor way to measure the health of your startup.

Focusing on the right users.

User engagement is the metric you need to measure to understand if people like your execution. After you launch it’s not about how many users signed up, it’s about how often they interact with your product. The users who use your site three times a day are exponentially more important than the ones who check it once a month.

Focus on increasing user engagement. It’s a better indicator of the health of your startup.

For more startup news, follow us on Twitter @startupfoundry or like us on Facebook.