Kaushik’s Impact Matrix – A Marketing Analytics Strategic Framework

For years, marketers have grappled with how to distill a massive number of available marketing analytics metrics down to something that is actually useful. Complicating this is the fact that what is useful varies by who is consuming the information, with need for information becoming increasingly strategic as one ascends from manager to director, to C-suite executive (CxO). This complexity impairs decision making in both business leaders and analytics leaders:

  • Executives fail to truly understand the potential and value of analytics, and thus do not make data-driven decisions; and
  • Analytics leaders fail to truly understand the bottom line financial impact of their work, resulting in analytical strategies that are uninformed by that bottom line, and driven instead by new tool features (in 2012: “Ooh! Multi-touch attribution!” / in 2015: “Ooh! Cross-platform attribution!”) and random expert recommendations.

The dilemma of how to simplify this complexity, to create sighted business and analytics leaders, spurred former Google Analytics head Avinash Kaushik to create a simple visual that absorbs the scale, complexity and many moving parts of the digital analytics universe.

Kaushik’s Impact Matrix helps one realize the extent to which analytics impacts the business bottom line today, and what one’s future analytics plans should accomplish.

Plotted on a graph, each cell contains a metric (online, offline, or nonline). Business impact is on the y-axis, and ranges from Super Tactical (the smallest possible financial impact) to Super Strategic (the greatest). The scale on the y-axis is exponential (you’ll notice the numbers in light font between Super Tactical and Super Strategic on the graphic below go from 4 to 10 to 24 to 68 and onward). This demonstrates that impact is not an incremental change – every step up delivers massively greater impact.

The x-axis represents time-to-useful, from Real-Time to Semi-annually.

impact-time-metrics-matrix-shell-sm

Why time-to-useful? While most data can now be collected in real-time, not all metrics are useful in real-time. For example, impressions can be collected in real-time and is useful in real-time (if actioned, impressions can have a super tactical impact). Customer Lifetime Value on the other hand takes a much longer time to become useful, and if actioned, it can have a super strategic impact on the business – tens of millions of dollars.

Here is a representation of these ideas on Kaushik’s Impact Matrix:

impact-time-metrics-matrix-framing_sm

Impressions can be used in real-time for decision-making by your display, video and search platforms (e.g., via automation). You can report Gross Profit in real-time, of course, but doing so is almost entirely useless. It should be deeply analyzed monthly to yield valuable, higher impact actionable insights. Finally, Lifetime Value will require perhaps the toughest strategic analysis, from data accumulated over months, and the action takes time to yield results – but those results are extremely useful.

In all, the Impact Matrix contains 45 of the most commonly used business metrics – with an emphasis on sales and marketing. The metrics span digital, television, retail stores, billboards, and any other presence of a brand you can think of. Note: You see more digital metrics because digital is more measurable.

Some metrics apply across all channels, like Awareness, Consideration and Purchase Intent. You’ll note the most critical bottom line metrics, which might come from your ERP and CRM systems, are also included.

Every metric occupies a place based on business impact and time of course, but also in context of other metrics around it.

Here’s a magnified view that includes the bottom left portion of the matrix:

impact-time-metrics-matrix-close-up_sm

[You can download an Excel version of Kaushik’s Impact Matrix at the end of this post.]

Let’s continue to internalize impact and time-to-useful by looking at a specific example: Bounce Rate. It’s in the row indicating an impact of four and in the time-to-useful column weekly. While Bounce Rate is available in real-time, it is only useful after you’ve collected a critical amount of data (say, over a week).

On the surface, it might seem odd that a simple metric like Bounce Rate has an impact of four and TV GRPs and % New Visits are lower. Kaushik’s reason for that is the broader influence of Bounce Rates.

Effectively analyzing and acting on Bounce Rates requires the following:

  • A deep understanding of owned, earned and paid media strategies.
  • The ability to identify any empty promises made to the users who are bouncing.
  • Knowing the content, including its emotional and functional value.
  • The ability to optimize landing pages

Imagine the impact of those insights; it is well beyond Bounce Rates. That is why Bounce Rate garners more weight than Impressions, Awareness and other common metrics.

When designating a metric a KPI, depth of influence is your foremost consideration.

Here’s the full version:

impact-time-metrics-matrix-complete-sm

[You can download an Excel version of Kaushik’s Impact Matrix at the end of this post.]

The value of voice of customer metrics is evident by their high placement in context of the y-axis. Take a look at where Task Completion Rate by Primary Purpose and Likelihood to Recommend are, as an example. They are high in the hierarchy due to their positive impact on both the business and the company culture – thus delivering a soft and hard advantage.

You’ll also note that most pure digital metrics – Adobe, Google Analytics – sit in the tactical bottom line impact. If all you do day and night is just those metrics, this is a wake-up call to you in context of your actual impact on the company and the impact of that on your career.

At the top-right, you’ll discover Kaushik’s obsession with Profit and Incrementality, which he asserts form the basis of competitive advantage in 2018 (and beyond). Analyzing these metrics not only fundamentally changes marketing strategy (think tens of millions of dollars for large companies); their insights can change your company’s product portfolio, your customer engagement strategies and much more.

The matrix also includes what is likely the world’s first widely available machine learning-powered metric: Session Quality, which you’ll  find roughly in the middle. For every session on your desktop or mobile site, Session Quality provides a score between 1 and 100 as an indication of how close the visitor is to converting. The number is computed based on a ML algorithm that has learned from deep analysis of your user behavior and conversion data.

The placement of each of the 45 metrics will help you add metrics that might be unique to your work. With a better understanding of the matrix, you are ready to overcome the two problems that broke our hearts at the start of the post – and do something super-cool that you did not think we might.

How to Operationalize the Impact Matrix for Your Organization

The impact matrix only works if you adapt it to your own organization. Here’s how.

1. Assess Analytics Program Maturity

Answer this simple question: What metrics are most commonly used to make decisions that drive actual actions every week/month/more?

Ignore the metrics produced as an experimental exercise nine months ago and the metrics you wish you were analyzing, but don’t currently. Assess the current reality.

Take the subset of metrics that actively drive action, and change the font color for them to green in the Impact Matrix.

For a large corporation with a multi-channel existence, here’s what the Impact Matrix looked like:

impact-time-metrics-matrix-analytics-program-savvy-sm

[You can download an Excel version of Kaushik’s Impact Matrix at the end of this post.]

The company’s offline marketing strategy spans television and other offline advertising, including retail.

You’ll likely recognize many of these metrics as the ones that your analytics practice produces every day. They represent the result of a lot of hard work by the company employees, and external analytics partners.

We are trying to answer the how much does the analytics practice matter question. You can see that more sharply now.

For this company most green metrics cluster in the bottom-left quadrant, with most having an impact of ten or under (on a y-axis scale of 1 to a ). There is one clear outlier (Nonline Direct Revenue – a very difficult metric to compute, so hurray!)

As every good consultant knows, if you have a 2×2 you can create four thematic quadrants. In our case the four quadrants are called Solid Foundation, Intermediate, and Advanced:

[You can download an Excel version of Kaushik’s Impact Matrix at the end of this post.]

For this company, the maturity of the analytics practice fit mostly in the Solid Foundation quadrant.

Is this a good thing?

It depends on how long the analytics practice has been around, how many Analysts the company has, how much money it has invested in analytics tools, the size of their agency analytics team, so on and so forth.

If they have a team of 50 people spending $18 million on analytics each year, over the last decade, with 12 tools and 25 research studies each year… You can now infer that this is not a good thing.

Regardless, the Impact Matrix now illuminates clearly that highly influential metrics are underutilized. These are the metrics  that facilitate deeper thought, patience and analysis to deliver big bottom line impact.

Recommendations:

  1. Conduct this exercise for your own company. Identify the metrics actively being used for decision-making. Which quadrant reflects the maturity of your analytics program? With the investment in people, process, tools, and consultants, are you in a quadrant where your bottom line impact is super strategic?
  2. Identify your target quadrant. In this instance the company could move bottom-right and then up. They could also move top-left and then top-right. The choice depends on business strategy and current people, process, tools reality. This should be obvious; you always want the Advanced quadrant lit up. But, you can’t go from Beginner to Advanced directly – evolution works smarter than revolution. If your Solid Foundation quadrant is not lit up, do that first.
  3. Create a specific plan for the initiatives you need to undertake to get to your next desired quadrant. You’ll certainly need new talent, you’ll need a stronger strategic leader (less ink, more think), you’ll need to identify specific analytics projects to deliver those metrics, and you’ll most definitely need funding. Divide the plan into six-month segments with milestones for accountability. The good news is that it is now, finally, clear where you are going AND why you are going there. Congratulations!
  4. Start a cultural shift. Share the results of your assessment, the green and black reflection of the current reality, with the entire company. Inspire each Marketer, Finance Analyst, Logistics Support Staff, Call Center Manager, and every VP to move one step up or one step to the right. If they currently measure AVOC, challenge them to move to Unique Page Views or Click-thru Rate. It will be a small challenge, but it will improve sophistication and, as you can see in the matrix, the impact on the bottom line.

Most companies wait for some Jesus-Krishna hybrid to descend from heaven and deliver a glorious massive revolution project (overnight!). These never happen. Sorry, Jesus-Krishna. Instead, what it takes is each employee moving a little bit up and a little bit to the right while the Analytics team facilitates those shifts. Small changes accumulate big bottom line impact over time.

So. What’s your quadrant? And, what’s your next right or next up move?

2. Align Metrics & Leadership Altitude.

People commonly believe that more data means better results. Or, that if an Agency is providing a 40 tab, font size 8, spreadsheet full of numbers that they must have done a lot of work – hence better value for money. Or, a VP wants two more histograms that represent seven dimensions squeezed into her one page dashboard. More metrics ≠ smarter decisions; the right metrics reaching the right person at the right time does.

To ensure this, we must consider leadership altitude (aka the y-axis).

Altitude dictates the scope and significance of decisions. It also dictates the frequency at which data is received, along with the depth of insights that need to accompany the data (IABI FTW!). Finally, altitude determines the amount of time allotted to discuss findings.

Managers have a lower altitude as they are required to make tactical decisions (impacting hundreds of thousands of dollars). VPs have a higher altitude, they are paid a ton more in salary, bonus and stock, because they carry the responsibility for making super strategic decisions – impacting millions of dollars.

Slice the matrix horizontally to ensure that the metrics delivered to each leader are aligned with their altitude.

impact-time-metrics-matrix-leadership-levels_sm

[You can download an Excel version of Kaushik’s Impact Matrix at the end of this post.]

VPs sit at decision making that is squarely in the Super Strategic realm – on our scale ~40 and higher. This collection of metrics power heavy decisions requiring abundant business context, deep thinking and will influence broad change. Analysts will need that time to conduct proper analysis and obtain the IABI.

You can also see that nearly all metrics delivered to the VPs arrive monthly or even less frequently. Another reflection of the fact that their altitude requires solving problems that will connect across orgs, across incentives, across user touch points, etc.

So. Are the metrics on your VP Dashboards/Slides the ones in Super Strategic cluster?

Or. Is your analytics practice such that your VPs spend their time making tactical decisions?

Below the VP layer, you’ll see metric clusters for slightly less strategic impact on the company bottom line for Directors. The time-to-useful also changes on the x-axis for them. Following them is the layer for managers, who make even more frequent, tactical  decisions.

The last layer is my favorite way to improve decision making: Removing humans from the process. 🙂

Recent technical advancements allow us to use algorithms – machine learning – to automate decisions made by metrics that have a Super Tactical impact. For example, there is no need for any human to review Viewability because advanced display platforms optimize campaigns automatically against this metric. In fact an expensive human looking at reports with that metric will only slow things down – eliminating the fractions of penny impact that that metric delivers.

Recommendations:

  1. Align metrics to altitude. Collect the dashboards and main reports created by your analytics practice. Cluster them by altitude (VP, Directors…). Identify if the metrics being reported to each leadership layer are the ones being recommended by the Impact Matrix. For example: Does your last CMO report include Profit per Human, Incremental Profit per Non-line Channel, % Contribution of Non-line Channels to Sales? If yes, hurray! Instead, if they are reporting Awareness, Consideration, Intent, Conversions, Bounce Rate, etc., your organization should ask itself why its CMO is using his or her valuable time making tactical choices? Is it a culture problem? Is it a reflection of the lack of analytical savvy? Figuring this out will go a long way toward solving the problem, and the above analysis will help identify core issues that are stymieing the contribution that data can make to smarter, faster, business success.
  2. Kick off a specific initiative to tackle automation. If data is available in real-time and useful in real-time, there are algorithms out there that can make decisions for humans. If there is a limitation, it is all yours (people, bureaucracy, connection points, etc.). For the other layers, action will depend on what the problem is. It could require new leadership in the analytics team, it could require a shift in company culture, or it could require a different engagement model across layers (managers, directors, VPs). One thing adjusting the altitude will certainly require: Change in how employees are compensated.

As you notice above, the strength of the matrix is in it’s ability to simplify complexity.

3. Strategically Align Analytical Effort.

One more slicing exercise for our matrix, this time for the analytics team itself.

Analytics teams face a daunting challenge when figuring out what type of effort to put into tackling the fantastic collection of possibilities represented in the Impact Matrix.

That challenge is compounded by the fact that there is always too much to do and too few people to do it with. Oh, and don’t get me started on time! Why are there only 24 hours in a day??

So, how do we ensure that each has an optimal analytical approach?

Slice the matrix vertically along the time-to-useful dimension…

impact-time-metrics-matrix-analytical-effort_sm

[You can download an Excel version of Kaushik’s Impact Matrix at the end of this post.]

For any metric that is useful in real-time, we have to unpack the forces of automation. Campaigns can be optimized based on real-time impressions, clicks, visits, page views, cost per acquisition etc. We need to stop reporting these, and start feeding them into our campaign platforms like AdWords, DoubleClick etc. With simple rules – ranges mostly – automation platforms can do a better job of taking action than humans.

If you are investing in machine learning talent inside your team, even narrowly intelligent algorithms they build will learn faster and surpass humans quickly for these simple decisions.

With the day-to-day sucking of life spirit reduced, tactical impact decisions automated, the analytics practice has time to focus on metrics that have a longer time-to-useful and need deeper human analysis to extract the IABI.

For metrics available weekly or within a few weeks, reporting to various stakeholders (mostly Managers and Directors) should adequately inform decisions. Use custom alerts, trigger threshold targets and more to send this data to the right person at the right time. For weekly time-to-useful metrics, your stakeholders have enough tactical context that you don’t need to spend time on deep analysis since the metrics inform the tactical decisions.

More role clarity, a thoughtful shift of the burden to the stakeholders, and more free time to focus on what really matters.

For where time-to-useful is in the month range, you are now truly heading into strategic territory. Reflect on the metrics there – challenging, strategic, Director and VP altitude. It is no longer enough to just report what happened, you need to identify why it happened and what the causal impact is for the why factors. This will yield insights that will have millions of dollars of potential impact on the company. That means, you’ll need to invest in ensuring your stories have more than just insights but also include specific recommended actions and predicted business impact. Amazingly, you’ll have just as much text as data in your output (that’s how you know you are doing it right!).

Finally, we have the pinnacle of analytics achievement. Our last vertical slice includes metrics that measure performance across customer segments, divisions and channels, among other elements. This is where meta-analysis comes into play, requiring even more time, with even more complex analytical techniques that pull data into BigQuery or similar environments where you can do your own joins, unleash R, use statistically modeling techniques (hello random forests!) to find the most important factors affecting your company’s performance.

The distribution of your analytical team’s effort across these four categories is another method of assessing maturity as well as ensuring optimal impact by the precious few analytical resources. For example: If most of your time is occupied by providing data to decision-makers for metrics in the Automate and Reporting vertical slices, you are likely in the beginner stage (and not having much impact on the business bottom line).

Recommendation:

  1. Conduct a time/effort analysis. Find an empty conference room. Project all the work your team has delivered in the last 30 days on the screen. Cluster it by Automated, Reporting, Analysis and Meta-Analysis. Roughly compute what percentage of the team’s time was spent in each category. What do you see? Is the distribution optimal? And, are the metrics in each cluster the ones identified by the Impact Matrix? The answers to these questions will cause a fundamental re-imagination of your analytics practices. The implications will be deep and wide (people, process, tools).

At the core of Kaushik’s Impact Matrix is the only thing that matters: the business bottom line. Using two simple dimensions, business impact and time-to-useful, you can explain simply three unique elements of any successful analytics practice. The reflections are sometimes painful, but bringing them to light allows us to take steps toward systematic improvement of our analytical practice.

Avinash has graciously provided an Excel version of the Impact Matrix on his blog Occam’s Razor for everyone to use. Good luck!

Recommend
  • Facebook
  • Twitter
  • LinkedIN
Share