Sunday Rewind: When A/B tests aren’t enough by Scott Castle

This week’s Sunday Rewind is a ProductTank San Francisco talk from Scott Castle, then VP at Sisense, about using data other than A/B tests and clickstreams to build products with longer, more complex user flows.

Scott starts by showing the UI for one of the products he was responsible for - highlighting the number of active UI elements, and the complexity and sophistication of the product. Analysing a simple clickstream of events doesn’t help him make sense of what his users are doing and he talks about blending different data sources to build an understanding of his customers and whether his product teams are building the right things to create customer success.

Scott explains a method he uses to understand how much each aspect of a B2B product affects his customer base. He tracks what product capabilities his customers are using within 60 days of a deal closing, and attributes the value of the deal to those features. He can then work out which features and capabilities are most valuable to which segments of his market.

Sometimes new features are added to solve a specific customer problem, or old features are replaced with improved versions, but the replacement doesn’t get adopted. Scott tells a story from his days at Periscope Data that explains how he dealt with this.

B2B products can take a long time to be rolled out across customers, so it can be hard to know if a usage delay is normal or a cause for concern. Adoption KPIs and goals need to be balanced to give a realistic expectation of your customer’s behaviour. Scott’s suggestion is to track time-to-adoption for various kinds of features and, over time, you’ll start to build up an idea of how long it takes customers to adopt different features.

User tenure with your product will determine how you make sense of their behaviour and their feedback, as different things will be relevant to new and long-term users. Scott recommends looking at the difference in time between when your users first started using a given feature, and when they most recently used that feature.

Watch the original talk: When A/B tests aren’t enough by Scott Castle