How Amazon craft their metric and wait, it's all loops? - Data and Decision #4

My observation how this company proposed new approach for picking metrics and why I think the world revolve around loops in this fourth edition.

Hello there!

The number of subscribers for this newsletter has grown to 6 people (Woahh :D), and I wish you could see how happy my face was when I saw this number. The fact that my weird and inconsistent posts can attract people is still beyond my imagination.

A heartfelt thank you to all of you, especially Robert from Win With Data (I’m a big fan of his work!).

Today, I'll keep this introduction short since I've included unusually lengthy content in our first insight. Let's dive right in!

---

2 Insights

How Amazon Formulate Their Metric - Another Approach? (Warning: long post)

I reread 'Working Backwards' again and stumbled upon something that I had overlooked in the past. Colin, the author of the book, mentioned how Amazon came up with 'Fast Track In Stock,' a metric they use as a proxy to measure the quality of Amazon's product selection.

See if you find something interesting.

When we realized that the teams had chosen the wrong input metric—which was revealed via the WBR process—we changed the metric to reflect consumer demand instead.

Over multiple WBR meetings, we asked ourselves, “If we work to change this selection metric, as currently defined, will it result in the desired output?” As we gathered more data and observed the business, this particular selection metric evolved over time from:

1. number of detail pages, which we refined to

2. number of detail page views (you don’t get credit for a new detail page if customers don’t view it), which then became

3. the percentage of detail page views where the products were in stock (you don’t get credit if you add items but can’t keep them in stock), which was ultimately finalized as

4. the percentage of detail page views where the products were in stock and immediately ready for two-day shipping, which ended up being called Fast Track In Stock.

You’ll notice a pattern of trial and error with metrics in the points above, and this is an essential part of the process. The key is to persistently test and debate as you go. (emphasis added)

“Ehm, okay, so what’s it all about?”

In short, the example above showcases another approach that we can use to craft and select a metric – a key skill for any analyst out there.

From my past experiences, I've come to realize that there are essentially two approaches we can take when creating and choosing metrics – the 'output/uncontrollable' route or the 'input/controllable' route.

In the first approach, we opt for metrics that are considered as output metrics, characterized by our lack of direct influence (as the term suggests, “uncontrollable”). Examples include “Total Products Sold” or “Monthly Revenue.”

For a clearer illustration: imagine your boss telling your entire department, “We need to increase our revenue per month!” and then walking away. What exactly would you do in the next 24 hours based on that directive? If you can't answer that straightforwardly, it's probably an output metric.

The second approach, on the contrary, is about selecting what's often referred to as an “input metric,” or as I prefer to call it, a “controllable” metric. As the name implies, this metric encompasses every variable that we can manage and influence. Examples include “Number of Articles Published” and “Percentage of Goods in Ready Stock.

Using the previous scenario: if your boss were to call the company's content manager out of the blue and say, “The number of articles published this month is too low!”, they would have a fair idea of what to focus on in the next 24 hours.

But I missed the 3rd option where we can mix these two approaches.

There's a choice that not only employs controllable metrics but also blends them with what I'd call an “output-filtered” component. Amazon's 'Fast Track In Stock' metric is an example of this hybrid approach. I haven't quite pinned down the perfect term for this, so for now, let's call it “controllable-but-filtered” (as you can probably tell, naming things isn't exactly my forte, so I apologize in advance).

I'm about to show you how incredibly useful this approach is. But before we dive in, let's talk about what sparked this realization.

By a twist of fate, I stumbled upon a similar concept while reading a book on my current list, Eliyahu’s “The Goal.”

Before I share an excerpt from the book, let me give you some context so you can truly grasp the profoundness of the upcoming discussion.

In a nutshell, Alex (the main character in the book), who leads a plant facing the threat of bankruptcy, suddenly comes to a striking realization: his plant, up until that point, didn't truly understand 'The Goal'... which, as it turns out, is quite simply to make money (obvious, right?).

With this revelation as his starting point, he embarks on a journey to unravel the deeper meaning of 'making money.' However, he struggles to translate this understanding into actionable steps for his manufacturing-oriented team. He's searching for a way to provide clear instructions to guide his team's efforts.

So, there you have it: a plant manager with the burning desire to generate profits, yet grappling with how to translate this aspiration into practical guidance for his team.

Below is a snippet of his conversation with Jonah, his mentor and consultant, whom he often turns to for advice on factory challenges (I've edited the conversation for clarity and ease of reading).

Jonah: “You see, there is more than one way to express the goal. Do you understand? The goal stays the same, but we can state it in different ways, ways which mean the same thing as those two words, ‘making money’.

Alex: "Okay, so I can say the goal is to increase net profit, while simultaneously increasing both ROI and cash flow, and that’s the equivalent of saying the goal is to make money.’’

Jonah: "Exactly. One expression is the equivalent of the other. But as you have discovered, those conventional measurements you use to express the goal do not lend themselves very well to the daily operations of the manufacturing organization. In fact, that’s why I developed a different set of measurements.’

Alex: "What kind of measurements are those?’’

Jonah: "They’re measurements which express the goal of making money perfectly well, but which also permit you to develop operational rules for running your plant. There are three of them. Their names are throughput, inventory and operational expense”. ’ 

For those of you readers who are engaged in manufacturing or accounting, this term might already ring a bell. However, pay special attention to the author's definition of throughput provided below.

Continuing…

Alex: "Those all sound familiar.”

Jonah: "Yes, but their definitions are not. In fact, you will probably want to write them down.’’ 

Alex: (With pen in hand) “Go ahead”.

Jonah: "Throughput, is the rate at which the system generates money through sales.’’ 

Alex: (Write it down word for word) "But what about production? Wouldn’t it be more correct to say—’’ 

Jonah: “No. Through sales—not production. If you produce something, but don’t sell it, it’s not throughput. Got it?”

Alex:  “Right. I thought maybe because I’m plant manager I could substitute—’’ 

Jonah: "Alex, let me tell you something. These definitions, even though they may sound simple, are worded very precisely. And they should be; a measurement not clearly defined is worse than useless. So I suggest you consider them carefully as a group. And remember that if you want to change one of them, you will have to change at least one of the others as well.’’

That brief exchange really got me thinking. I suddenly grasped the significance of including the "sales" component, the one that Jonah strongly advocated for, instead of Alex’s "production," in the definition.

But why is it so crucial?

Adding “sales” in the definition links the action Alex takes – production – with a paramount filter: the market. This single adjustment completely shifts Alex's focus from mere "efficiency," as outlined in the book, and eventually leads him to uncover the concept of bottlenecks and other pivotal changes.

By incorporating the market filter, his plant transforms from struggling to becoming the top-performing facility in the entire company.

"Alright, Erald. This is all fun and interesting, but what can I actually take away from this?"

There are a few truly useful things that we can gather from this.

You see, each of the first two approaches has a fundamental flaw.

Focusing solely on an output metric doesn't provide any valuable insights for your company's operations and strategy. Conversely, fixating only on what you can control is also a misstep – you still need to assess whether the chosen metric effectively measures your efforts toward the goal.

Amazon tackles this by integrating a series of subtle yet actionable filters that are effective in sidestepping the trap known as "Goodhart's Law." This perspective has dramatically shifted my mindset regarding how I typically propose metrics within my division. I'm planning to internalize this, put it to the test, and circle back to you all with my findings.

"But hold on, isn't this just a 'quality metric' with extra steps?"

Those of you in the data field might be thinking, "Doesn't this concept resemble the pairing of both quality and productivity metrics?" I was initially tempted to say yes, but there's a subtle distinction.

In theory, combining productivity and quality metrics might suffice, but in practice, most teams tend to disregard the quality element and focus solely on maximizing productivity metric. As someone who's heavily driven by data (a "DaT4-drIv3n" person), my most agonizing experience has been witnessing how quality metrics often get overlooked. It all makes sense considering that, during that period, all KPIs were productivity-based, leading people to respond to incentives and ignore the quality part.

By bundling both aspects together using “controllable-but-filtered” approach, it becomes harder for the team to neglect the quality facet.

Remember, your actions still need to be filtered through the outcomes. Or, to put it another way: Are you certain that your controllable metric (the action you took) yields results?"

Topic 2: Wait, it’s all “loops”?

Whoa, my previous insight was quite the read. So let's keep this one brief.

My curiosity of this topic springs from my 'dumb question': What if I told you that, instead of a funnel, the world around us is actually a loop?

Okay here’s few reason.

• [Motivational Reason] Viewing the world as a loop unveils a myriad of actions we can take to reach our goals. With this perspective, we become more inclined to seek alternative paths to achieving our objectives. Struggling with your personal health target? Consider exploring interesting podcasts that enhance your jogging experience, subsequently boosting your consistency for doing it every day. Got this from personal experience.

• [The Way the World Operates?] You're likely familiar with Pareto's Law, which observes how a minority within a group holds the majority of the output. This implies a crucial notion: winners possess more resources, allowing them to secure further victories compared to their competitors. It's a 'loop'-inspired way of thinking.

This thought-provoking post from CommonCog was originally intended to explore crafting effective questions for acquiring skills from experts. Yet, Cedric's talent shines through as he introduces us to the ongoing 'loops vs funnel' debate in the realm of growth marketing.

It opens a window to understanding each approach. Definitely a must-read!

---

1 Big Question

“How to cultivate “glue person”-skill as a data analyst in any company?”

---

1 Quote

“Take a simple idea and take it seriously.”  - Charlie Munger. 

This quote will set the theme of all my writing moving forward, at least for the next 5 years. My career and personal life has been filled by Munger’s wisdom but I felt like I didn’t fully internalize this concept to take a simple idea very very seriously. Well, that’s for another day.

Until next time,