Data without intelligence is dangerously misleading. Focus on the right users.

Startups love numbers. Data is king in the A/B testing world. Numbers allow entrepreneurs to quickly quantitative decisions with hard data. In this data-driven world it’s easy to forget that data without intelligence is dangerously misleading.

For example:

Back during World War II, the RAF lost a lot of planes to German anti-aircraft fire. So they decided to armor them up. But where to put the armor? The obvious answer was to look at planes that returned from missions, count up all the bullet holes in various places, and then put extra armor in the areas that attracted the most fire.

Obvious but wrong. As Hungarian-born mathematician Abraham Wald explained at the time, if a plane makes it back safely even though it has, say, a bunch of bullet holes in its wings, it means that bullet holes in the wings aren’t very dangerous. What you really want to do is armor up the areas that, on average, don’t have any bullet holes. Why? Because planes with bullet holes in those places never made it back. That’s why you don’t see any bullet holes there on the ones that do return.

From Kevin Drum’s article on MotherJones.com

I see a lot of startups focus solely on increasing the number of users they have. I’ve received pitches from startups that say things like “Over 5,000 users have already signed up!”, and I’ll say “That’s fantastic but how many people are actively using your service?”. Total users are just bullet holes in the wing.

The right way to view total users

Looking at total users is a useful way to determine demand for an idea. If you haven’t launched yet and you already have over 1,000 users sign up, you can start gauging the market. Total users can help you quantify a market but it’s a poor way to measure the health of your startup.

Focusing on the right users.

User engagement is the metric you need to measure to understand if people like your execution. After you launch it’s not about how many users signed up, it’s about how often they interact with your product. The users who use your site three times a day are exponentially more important than the ones who check it once a month.

Focus on increasing user engagement. It’s a better indicator of the health of your startup.

For more startup news, follow us on Twitter @startupfoundry or like us on Facebook.

  • Alex4444

    I hadn’t thought about it in those terms before. Thanks Paul!

  • Very interesting; definitely makes me want to think twice whenever I use a set of data to make an assumption.

  • Couldn’t agree more. We are focusing our metrics only on active (signed in in last 30 days) users, and also looking at actions/day as an engagement metric.

  • The problem is most companies don’t know how to track “user engagement”. Its not a number that something like Google Analytics can give you. The reason people trip over these irrelevant numbers such as registered users and page views is because thats the only data they know how to actively track. Dave McClure once said, “User Engagement is a depth, not a breadth, metric.” Yahoo has also done some interesting research on the topic: http://research.yahoo.com/node/3426

    User engagement is really a mix of a lot of different metrics weighted differently for each business. Actions, context and motivation play a big part in engagement. To understand how your users are engaging with your company you need to understand exactly how they are using your product. Which features are they using the most? Do your users even know some features exist? Who are your “power users” or evangelists? Which users were lost because the on-boarding process was rough? Page views won’t give you these answers.

    Once you’re tracking the right things you’ll be able to quickly spot what needs improvement. Then you’ll ask: how do I get my users to engage more, tell their friends and come back tomorrow? How do I motivate users to behave in a way that is beneficial to both of us?

    I believe the next step is to leverage “gamification” to measure engagement and motivate users. That is why I started http://www.IActionable.com, to help companies understand how gamification can take their user engagement to the next level.

    • Anonymous

      Well thought out points (and nice hustle with the link at the end). Thanks for enriching the conversation!

    • AnandP

      Precisely why I made http://www.talkhotels.com an invite-only network. I want my members to invite people who they feel will make quality contributions, which in turn benefits everyone involved.