Innovation Accounting

So, we're getting ready to ship our first technology MVP.  We've done some concierge and pen and paper MVPs.  But now, its time to tighten the test-learn-iterate loop.  I've read Eric Ries' The Lean Start-up twice already.  But I was confused about the details of how I would measure what was working and what was not in our product.  One of the early things we want to learn is what information is important to a consumer when making a decision we are trying to help them with.  And also, what information is not important to that decision.  I was trying to come up with ways to measure each data element we added to the experience and somehow deduce it's importance to the consumer.

I had some ideas but none seemed that clear.  So, I went back to the book, particularly Chapter 7, which deals with measurement.  Halfway through the chapter, the answer hit me like a ton of bricks.  Determining what is important and what is not will be much easier than I was thinking thanks to innovation accounting.  Here's my summary of how it will work.

• First, we'll need to establish our overall funnel metrics.  This is pretty easy.  We'll track as cohorts each week.
• Second, we'll need to be able to support A/B testing.
• Third, when it's time to ship a feature, we'll ship it as an A/B test.  Some users will get the new feature; others won't.  We'll also ship one feature at a time to help isolate cause and effect.
• Fourth, we'll measure the funnel metrics for the two samples and determine what impact the new feature has on them.  If the new feature has a positive effect, great.  We'll keep the feature and continue to tweak it.  But if the feature has no effect or a negative effect, then we'll pull it back out of the product and move onto the next idea on the backlog.

This is a shift in the way I've done product development in the past.  Here are some of the ways it is different:

• Treat every new feature as a test that needs to justify it is staying in the product.  In the past, I've shipped new features but never asked if they should stay in the product once shipped.
• A/B test everything.
• If a feature is not having a noticeable positive impact on your top-line metrics, then its a distraction to the team and the customer and should be pulled.

I'm super excited to implement this new way of thinking.  I had a real hard time getting sleep last night as I kept thinking about all of the potential this methodology holds for quickly moving the product forward.

Stay tuned, we're shipping soon.