Innovation Accounting

by Scott Austin October 18, 2012

Innovation Accounting

So, we're getting ready to ship our first technology MVP.  We've done some concierge and pen and paper MVPs.  But now, its time to tighten the test-learn-iterate loop.  I've read Eric Ries' The Lean Start-up twice already.  But I was confused about the details of how I would measure what was working and what was not in our product.  One of the early things we want to learn is what information is important to a consumer when making a decision we are trying to help them with.  And also, what information is not important to that decision.  I was trying to come up with ways to measure each data element we added to the experience and somehow deduce it's importance to the consumer.

I had some ideas but none seemed that clear.  So, I went back to the book, particularly Chapter 7, which deals with measurement.  Halfway through the chapter, the answer hit me like a ton of bricks.  Determining what is important and what is not will be much easier than I was thinking thanks to innovation accounting.  Here's my summary of how it will work.

  • First, we'll need to establish our overall funnel metrics.  This is pretty easy.  We'll track as cohorts each week.
  • Second, we'll need to be able to support A/B testing.
  • Third, when it's time to ship a feature, we'll ship it as an A/B test.  Some users will get the new feature; others won't.  We'll also ship one feature at a time to help isolate cause and effect.
  • Fourth, we'll measure the funnel metrics for the two samples and determine what impact the new feature has on them.  If the new feature has a positive effect, great.  We'll keep the feature and continue to tweak it.  But if the feature has no effect or a negative effect, then we'll pull it back out of the product and move onto the next idea on the backlog.

This is a shift in the way I've done product development in the past.  Here are some of the ways it is different:

  • Treat every new feature as a test that needs to justify it is staying in the product.  In the past, I've shipped new features but never asked if they should stay in the product once shipped.
  • A/B test everything.
  • If a feature is not having a noticeable positive impact on your top-line metrics, then its a distraction to the team and the customer and should be pulled.

I'm super excited to implement this new way of thinking.  I had a real hard time getting sleep last night as I kept thinking about all of the potential this methodology holds for quickly moving the product forward.

Stay tuned, we're shipping soon.

Scott Austin
Scott Austin