10 tips for using OKRs effectively

I’m back after a long hiatus from writing.

We are in planning season at Digit. Like a number of other technology companies, we use OKRs. I was puzzled by the wide range of OKR effectiveness at different companies. My study of the nuances of different approaches led to the conclusions outlined in this note. OKRs are incredibly useful but it takes a long time for OKR usage to become effective. Shortcut your company's learning process by understanding and implementing the following ten ideas. If you find them helpful, drop me a note.

What are OKRs?

OKRs stands for Objectives and Key Results. They capture two things:

  1. Objective: The goal a company wants to achieve ("What")

  2. Key Results: The path to achieving that goal ("How")

OKRs are an invaluable management tool for organizations of all sizes. Used properly, they allow an organization to unlock four superpowers:

  1. Focus and commit to priorities.

  2. Align and connect for teamwork.

  3. Accountability through tracking.

  4. Stretch to achieve the impossible.

[Measure What Matters, John Doerr]

OKRs are a tool. The act of using OKRs does not guarantee Google-like performance, innovation, or market dominance. OKRs are not a strategy, nor do they make up for lack of strategy. OKRs force a company or organization to think rigorously, to hold themselves accountable, to stretch, and to learn and grow. Persistent usage over time guarantees increased focus, alignment, and execution. Here are 10 ideas to use them more effectively 👇🏽

10 Tips for using OKRs effectively

1. Objectives must be Big and Motivating

A great, challenging goal is inspiring. It brings teams together. It ignites that thing in us that led to human spaceflight and exploration of unknown frontiers. Inspirational objectives are critical at the company level, since company OKRs cascade down to teams and individuals. Everyone should feel proud to accomplish their objectives and together achieve a higher goal.

Objective: Win the Super Bowl

KR1: Passing attack amasses 300+ yards per game. 

KR2: Defense allows fewer than 17 points per game. 

KR3: Special-teams unit ranks in top 3 in punts return coverage.


2. KRs must be measurable

There are three important components of a key result:

  1. The metric by which you will measure progress.

  2. Where you are starting and what the goal is.

  3. When you'll be done. The time is assumed to be end of the execution period (quarter or half) if not mentioned explicitly.

For example, let's say you want to reduce your AWS costs by improving server efficiency.

Objective: Reduce AWS costs.

KR: Increase server utilization.

How will we know whether we have achieved our OKR or not? When should we stop working on this or work harder? The OKR could be rephrased to make it clearer:

Objective: Reduce AWS costs from $100 to $80.

KR: Increase server utilization from 65% to 80% by the end of the quarter.

3. Use binary KRs sparingly

A binary KR is one that is either done or not done. For example:

Objective: Reduce AWS costs from $100 to $80.

KR: Turn on auto-scaling for all services by the end of the quarter.

Binary KRs are measurable — they were either accomplished or not — but usually need to be combined with quality counter-metrics to be effective. More on metrics and counter-metrics below.

Another example of binary KRs are "Ship KRs". They are often used when rolling out new features. Without baseline data, there is no way to measure or predict how a feature will perform. The team can put in work to predict outcomes, but sometimes, it is easier to ship something and see the impact. Ship KRs have their place, use them with caution.

4. All Key Results must have dashboards

The natural consequence of measurability is the scoreboard. Every measurable KR should have an associated dashboard. These need not be fancy — an excel sheet is sufficient to get started.

Some results are only measured fully in retrospect or take a while to get an accurate read. Retention is an example of such a measure — we don't know upfront which user will churn. In such cases, teams should use alternate approaches, such as:

  1. Create a proxy "operational" metric that correlates well with the goal metric. 1-day retention can be used for 30-day retention if we find that the bulk of churned users churn within a day.

  2. Use a forecasted or predicted number as the operational metric. Monitor the predicted metric to keep the error within bounds.

5. Key Results must be exhaustive

Another common trap that leads to a feeling of inadequacy at the end of the quarter is missing an objective despite accomplishing its KRs. Imagine we take on a feature activation goal that ladders to some critical company objective.

Objective: Increase usage of feature X from 100 DAU to 200 DAU.

KR1: Improve activation funnel efficiency from 85% to 90% 
by August 1.

KR2: Improve feature X retention from 90% to 95% 
by September 1.

We get to the end of the quarter, both KRs are green, and feature usage hasn't gone up at all.

What if the lack of feature X had more to do with how you marketed it? If the problem is at the top of the funnel, increasing funnel efficiency is unlikely to achieve the goal.

Missed KRs are abundantly evident in hindsight, but much harder to think of ahead of time. One approach is to ensure that the goal (or the first KR) captures the intended outcome (100 DAU to 200 DAU) and let the team add KRs along the way if enough progress is not being made.

6. Pair Metrics with Counter-Metrics

There is natural pressure and temptation to achieve progress at all costs. There have been innumerable examples in the corporate world, with Enron and Wells Fargo being the most recent, where the drive to achieve a particular outcome, such as "Open more accounts," compromised the company's core values.

To guard against such adverse outcomes, you can pair metrics with counter-metrics. For example, Wells Fargo could have paired "number of active accounts" with defensive metrics such as "each account must be active," "each account must have a certain balance," "no more than X accounts per person in a week."

In product-feature land, there is a natural pairing of growth-based KRs and quality-based KRs.

KR1: Feature X will have 100 users by the end of the quarter.

KR2: 7-day retention for Feature X will be 90%. 

KR3: P0 and P1 bugs will be closed within a day.

7. Distinguish between Committed and Aspirational OKRs

OKRs are scored on a scale of 0.0 to 1.0. What's a "good" score? Many tech organizations believe the answer is 0.7, based directly on Google's scoring approach [whatmatters.org — how to grade OKRs]

However, the Google scoring system applies to Aspirational OKRs — typically Big, Hairy, Audacious Goals that you don't know how to hit. You know it's going to be hard, and you're likely to fall short. But if you don't reach for the moon, you're never going to get there. The purpose of aspirational OKRs is to stretch you, to motivate you to solve more significant problems and to think differently.

In contrast, some OKRs are expected to be delivered in full. Doerr calls these Committed OKRs. For the business to succeed, for cross-functional teams to depend on each other, these goals must be achieved in full. You are pushing for complete success, not setting aspirational visions. The expected score for these OKRs is 1.0, with low variance.

Distinguishing between committed and aspirational OKRs is essential. If a team biases too much toward commitment, they may not stretch enough. If you go all aspirational, you may not hit any goal. Leadership must make deliberate choices about how their company should think about their OKR bias.

My current personal leaning is to bias toward committed OKRs for nearly everything, with one or two aspirational OKRs for things that are truly game-changing.

8. Cascade OKRs up, down, and laterally

High functioning organizations speak the language of OKRs. When you depend on another team to hit your goal, you should expect to see that dependency in the other team's OKRs. If you don't, your likelihood of failure has increased significantly.

Similarly, company Key Results often become Objectives for organizations within the company. Organization Key Results become Objectives for individual teams.

OKRs can also cascade upward. For example, a team could discover that they have no way of hitting their goals, given their level of staffing. They refuse to take on the OKR, which in turn changes the OKRs above them, and so on.

This cascade of OKRs through the organization is a critical step of aligning the entire company. While it may sound chaotic in theory, the company or organization should quiesce in a week or two.

9. Personal OKRs are powerful. Use them to accelerate your career

Company and Team OKRs drive organizational alignment and focus. You can create your personal OKRs too. While some will inevitably be congruent with your team's OKRs, there could be others that are about making progress in your career or growing as an individual. Personal OKRs clarify your priorities with your manager and your peers. They hold you accountable for accomplishing what you set out to do.

I think of Personal OKRs as writing my self-review, six months ahead of time. I have been creating personal OKRs, putting them into a document, and sharing them with my manager for about six years. They have helped me be clear with my manager about what constitutes success and failure, what is paramount in their minds, and keeps me accountable for making progress. I strongly recommend doing Personal OKRs and taking charge of your career.

When writing personal OKRs, you must focus on outcomes, not activities. This trap is avoided at the team and company level by making all KRs measurable. However, when looking at personal OKRs, we start using language like "Participate in interviewing" or "Be a part of X team." Consider rephrasing into outcomes instead, such as "Hire 3 engineers by the end of quarter" or "Conduct an average of 1 interview per week."

10. Prefer a small number of tightly focused OKRs to a long list

OKRs should add focus. A large number of OKRs, whether individual or team, imply a lack of focus. When push comes to shove, which OKR is going to drop? Don't bother adding OKR priorities — that is simply a bandaid over the root cause: you have too many OKRs.

OKRs should also represent commitment. Picking one OKR means that we cannot do something else. In your OKR discussions, you could include a list of things you chose not to do to illustrate an effort to add focus.

Bonus: Effective OKR usage takes years

If you're getting started with OKRs, know that your initial implementation will suck. You may have too many, or they may not be measurable, chaos may reign as OKRs cascade across the organization. Know that there is no pinnacle of OKR usage — Google struggles with them as much as a series-A startup. Their struggles are different, and they've had more than two decades to get it right. But keep at it and make your version of OKRs better over time. Be patient.


This note is a combination of practical experience from a decade of practicing OKRs and Goals at Google and Facebook, and reading through three books:

  1. The Practice Of Management — Drucker

  2. High Output Management — Grove

  3. Measure What Matters — Doerr


Many thanks to Leo Shklovski, Lily Zhang, Jules Walter, DeVaris Brown, and Ethan Bloch for reviewing early drafts of this work and providing invaluable feedback.

Joining Digit

From the non-COVID news corner

I've joined Digit as Chief Product Officer.

We are living through unprecedented times. Humanity is collectively battling an enemy that we cannot see, one that doesn't recognize national boundaries, and kills without remorse. The economic impact is horrendous. Nearly 6 million people have filed unemployment claims in the US in the last week alone [source] — a number never before seen in our country.

When people lose their livelihoods, they need cash to survive. Emergency funds are built for hard times, such as the ones we are experiencing right now. However, saving money, when you're living paycheck to paycheck, is incredibly hard. Human psychology works against you. We are wired to think in the present and to value immediate rewards. We need tremendous will power to look at every paycheck, and put some money away for a rainy day.

Digit started as an app to help people save. You connect your checking account to Digit, it analyzes your cash flow, and it intelligently moves your money to a Rainy Day Fund. You don't think about saving money, but you're saving. It is magical. Nobody wants to lose their job, but if it happens, at least you have some money saved up.

Digit's mission is to make financial health effortless for everyone. Our app supports saving for any goal, not just a Rainy Day. We help you automatically pay off your student loans. We help you automatically pay credit cards in advance to reduce your debt. The list of features is long and growing. We want people to feel less anxious about money, knowing that Digit is there to help them become and remain financially healthy. If the worst happens, and you lose your job because of a global pandemic, Digit is there to help.

Joining a company is a big decision. Being genuinely in love with the mission is a requirement, but it is not sufficient. Work is a large part of life. I believe in working with great people — who have done amazing things, are smart and driven, and above all, are exceptional human beings.

I have known Ethan Bloch, Digit's founder and CEO, since 2017. I have always admired Ethan's leadership — he stays true to his values and the mission, and does not hesitate to make hard calls. He has been a friend and someone I seek out for advice and counsel. Over time, he has built an exceptional leadership team with Michael Murray, Vishwas Prabhakara, Carolyn Satenberg, Karthik Hariharan, and Samar Shah. I'm thrilled to join this crew, along with Ryan Nier, our new head of Legal and Compliance, and call them colleagues and friends.

Digit is a fantastic company with great people who are dedicated to the company's mission. We're growing, and we're hiring, virus be damned. We are looking for product designers and product engineers who love building products that help people. If you're excited about our mission to make financial health effortless for everyone, come join us!

Product Designer: https://boards.greenhouse.io/digit/jobs/2004653

Software Engineer, Product: https://boards.greenhouse.io/digit/jobs/922676

Welcome To Remote Work

The COVID-19 Watershed Moment

The Trigger

As of March 2nd, 2020, the coronavirus outbreak has officially hit the tech sector. The virus, SARS-CoV-19, causes the disease, COVID-19. There are several different strains of the virus at this point, as it mutates at an average of about two mutations per month, resulting in several different strains in different places. [source]. There is a rash of cases in the Seattle area, with a smaller number reported in a host of other places, including California and the Bay Area.

COVID-19 Global Cases, ex-China. Data by Johns Hopkins CSSE

Elad Gil published an excellent post about the impact of the virus on tech companies, including detailed but readable background information on the virus and it's spread. Some useful Twitter accounts to follow are in the resources section below. Most important things to remember (summarized from Elad's post):

  1. We will likely see a widespread outbreak of COVID-19. However, most cases will be mild, and most people will recover.

  2. Symptoms manifest between 3-14 days from exposure, including fever, cough, and shortness of breath. [Mayo clinic]

  3. Case Fatality Rate (CFR: deaths / people infected) is estimated at 2-3% but could be much lower, with better healthcare. People over 60 have a higher risk, as do patients with existing cardiovascular disease, diabetes, chronic respiratory disease, hypertension, and cancer. [source] There have been no reported fatalities in infants and children. [source]

  4. R0 value estimate is between 2 and 3. Every infected person will likely infect 2-3 more. In contrast, flu R0 is about 1.5. [source]

Quarantine and focused personal hygiene appear to be our best defenses against the virus at this point. Use the WUHAN mnemonic as a reminder:

W - Wash hands with soap frequently. [YouTube U - Use a mask if you are sick and infected. H - Have your temperature taken daily if you have been exposed. A - Avoid large crowds and gatherings. N - Never touch your face with your hands.

The Response

Tech companies are starting to take action to protect their employees. I expect this trend to continue and accelerate over the next week. The responses generally follow the same principles:

  1. Minimize or cancel all work-related travel.

  2. Cancel large conferences and gatherings.

  3. Stay home if you exhibit any symptoms or believe you have been exposed to COVID-19.

  4. Working from home is encouraged for everyone and mandatory for critical employees.

  5. Interviews are over video instead of in-person.

Sources: Stripe's public post, Twitter's post,Coronavirus company response tracker

The Shift To Remote

COVID-19 represents a watershed moment for how we work.

Tech companies have been trying to figure out how to move more of their workforce to remote, to take advantage of talent pools outside of major hubs like San Francisco, New York, and Seattle. Most companies that start in one location advance step-by-step, moving toward a hub model, and some like Stripe, move to full remote in addition to having a central hub.

After the outbreak of the virus, no tech company has a real choice. Every company will become a remote-work company out of necessity. Leaders need to face the reality of not having everyone in the office — they should be proactive and not get caught flat-footed. Here are a few things to consider:

  1. Async Remote vs. Sync Remote. If your current culture is set up to work in one office, within one timezone, you should maintain the concept of "working hours" where everyone is present on Slack or Teams. Jumping to fully async remote, where everyone works any hours they like, might be too disruptive for your company's flow.

  2. Check your tools. Do you have enough zoom licenses? How are you going to do 100 people conferences remotely? Do you have a way to do collaborative whiteboarding? Are you capable of visually providing design feedback?

  3. Dry run. If your company isn't going full remote right away, you can still mandate staying home for a few days at a time, to work out the kinks. Do you have a process to deal with emergencies and outages? Do you have phone numbers in an easily accessible directory?

  4. Preserve culture. Every company has its customs and traditions, from Monday donuts to Friday beers. People celebrate achievements together, have ways of communicating and collaborating. Art — ranging from murals to laptop stickers — gives us a constant reminder of belonging and shared values. How will you preserve this when you don't have a common place to meet? For example, Fil Fortes uses a green screen to change up the background and keep things fresh.

Good books on remote work include Jason Fried and David Heinemeier Hansson's Remote: Office Not Required. The book not only makes a strong case for remote work (not required at the moment), but has several good tips, learned through experience.

Self and Family Care

There is a lot of information about COVID-19, and it's evolving fast. The consensus at this time, for yourself and your family, seems to be the following:

  1. Be prepared for a self-quarantine of 14 days or more. Stock up on food. ThePrepared has an excellent guide to getting food. Note that a lot of vendors are running out due to the increased demand. There is a distinction between food that will last for a decade (of which there seems to be short supply) and food that will last for a year (which is available in most grocery stores and Costco). The sooner you buy, the better.

  2. Be prepared for medicine shortages. Supply chains will get affected, and there could be runs on common drugs. Get a 90-day supply of your medication.

  3. Wash hands regularly. Regular hand washing (say ABC twice) is one of the best defenses against the virus spreading. The primary entry points are eyes, mouth, and nose, but we touch them with our hands roughly 23 times per hour. Hand sanitizers are only useful if they are 60% alcohol, and you rub your hands for 20 seconds or more. They're also out of stock (in the Bay Area.)

  4. Avoid large gatherings. Regularly wiping down commonly used surfaces, like door handles, will help prevent the spread of the disease in schools and other shared places.

I hope this has been helpful, and everyone stays healthy and safe.


Company plans

Twitters to follow

4 Interesting Reads

February 2020

I received positive feedback and encouragement for my previous 4 Interesting Reads post. Here is the next one. Feedback is always welcome, please reach out.

Making Uncommon Knowledge Common by Kevin Kwok


Rich Barton has founded three companies valued at over a billion dollars: Expedia, Glassdoor, and Zillow. In the linked article, Kwok explains Barton's central strategy: take public knowledge that is hard to obtain for historical or structural reasons, make it easily accessible, and as a result, own customer demand. When looking for flights or hotels, you start with Expedia. When you're looking to sell your house, your journey begins at Zillow with its Zestimate. This is data that has been converted to content that people want. All this "content" substantially lowers customer acquisition costs (CAC), the bane of a startup's unit economics. The takeaway for product people: cheaply create high quality, personalized content, and use it to drive down CAC, and provide value.

The Limits Of High Speed Rail by Ivan Rivera


I am fascinated and genuinely interested in the engineering problems faced in building things that are world record holders - the fastest car or train, the tallest building, the largest dam. For this reason, I could not put down Adrian Newey's autobiography How To Build A Car, about his journey in the world of Formula 1 racing, and the nonstop engineering ingenuity to build some of the world's fastest cars. In a similar vein, Rivera's article goes into detail about the engineering required to break the world record for the fastest train — SNCF's TGV V150 (named because 150m/s == 540km/h, the target speed) clocked 574.8 km/h on April 30, 2007. He goes into a lot of fascinating detail around wheel-rail contact points, why diesel-powered trains can't really go beyond 240 km/h, the role of aerodynamics, and the evolution of the pantograph-catenary contact (for electrical power transmission). I firmly believe that we advance the state of all human inventions and knowledge when we push the limits of what we can achieve, and for this reason, studying record-breaking inventions is worthwhile.

The New Business Of AI by Martin Casado and Matt Bornstein


The stellar people at Andreesen-Horowitz write incredibly insightful commentaries on entire industries: find the thing you're interested in (SaaS? Marketplaces? Crypto?) and follow the relevant a16z partners. Casado and Bornstein's post on the fundamentals of AI startups is packed with insights and hard to compress further. They posit that AI startups have lower gross margins (50-60%) compared to traditional SaaS businesses (60-80%) because of two reasons. 1) High cloud costs due to training, retraining, and inference over large data sets. 2) Human-in-the-loop costs, both for high-quality data (cleaning, labeling) and for checking the AI's results. AI startups have a harder time scaling because edge cases are much harder to deal with - there is a very long tail of edge cases due to the sheer volume of high dimensional space. Finally, they have weaker defensive moats because a lot of model development is being done in academia, or is open-sourced, and data scale effects are weaker than people think. I believe we are still in AI frontier land: new discoveries, new tools, new rules. Hence, understanding the business of AI is as vital as grokking the technology, and makes this article worth the time.

Group Chat: Group Stress by Team Basecamp


One of the most significant challenges of today's Slack based tech work environment is that there is no time to think. I don't believe Slack is the root cause; it is somewhere in between an enabler and an accelerant. But, there are too many channels to follow, lots of FOMO about stuff that does not matter, shallow thinking, and fast responses, no preservation of context, and forced ephemerality. The linked article (obviously biased, Basecamp is a competing product), lists out all these problems in greater detail. The central point, which I wholeheartedly agree with, is that "chat attacks attention, and severely hinders deep work." If you're in a Slack based environment and have figured out a way to do deep work without constant interruption, kindly write a post about it and help out the world (and let me know!)

Bonus: The Man In The Arena by Theodore Roosevelt

Doris Kearn Goodwin's book Leadership is a fantastic read about how 4 of America's great leaders (Johnson is controversial) suffered and dealt with significant personal hardship, and how it shaped their presidency and their leadership. Theodore Roosevelt is my favorite leader because of his bias toward action, his values and morality centered around always doing the right thing, his boldness, and his courage. I come back to this quote often, whenever I'm feeling low about my work, or overly criticized, or when an idea that I thought was going to change the world turns out to be a dud, or when I am scared of taking a leap. I hope it helps you as much as it has helped me.

It is not the critic who counts; not the man who points out how the strong man stumbles, or where the doer of deeds could have done them better. The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood; who strives valiantly; who errs, who comes short again and again, because there is no effort without error and shortcoming; but who does actually strive to do the deeds; who knows great enthusiasms, the great devotions; who spends himself in a worthy cause; who at the best knows in the end the triumph of high achievement, and who at the worst, if he fails, at least fails while daring greatly, so that his place shall never be with those cold and timid souls who neither know victory nor defeat.

— Theodore Roosevelt, Citizenship in a Republic

3 Frameworks For Making Complex Decisions

Life is full of complex decisions: capital purchases, such as a car or a house, planning a vacation, choosing a new job, picking a product strategy, prioritizing roadmaps, hiring someone. Complex decisions have several shared traits: the list of options is often extensive, evaluation criteria are ill-defined, the outcomes are hard to predict, input data is unavailable or incomplete. Humans understand large systems by building mental models, which are more straightforward than the reality they represent. Mental models are a great thing: they allow us to make progress without getting bogged down in every little detail. But they also have their flaws. Most notably, human cognitive biases, our failures to think and communicate clearly, lead us to sub-optimal decisions.

Psychologists and behavioral economists have spent a considerable amount of time and mental energy dedicated to understanding the human decision-making process. By systematically understanding our cognitive biases and flaws, smart people have come up with frameworks to counteract their ill effects. Using these frameworks can lead to decisions with better eventual outcomes., how we fail, and how to make decisions that lead to better results. Amos Tversky, Daniel Kahneman, Richard Thaler, Dan Ariely, and Chip Heath have done seminal research in this field — and distilled their ideas into highly readable books. Thinking FastandSlowNudgePredictably Irrational, and Decisive are amongst my favorites. I strongly encourage you to read these to gain a deeper understanding of the field.

In this post, I present three practical frameworks to improve decision- making in different contexts. Frameworks are hard to understand in the abstract. Just reading theory leads to a shallow understanding of how to apply them in practice. To make things more concrete, I use two practical problems that I have solved using some combination of these three frameworks.

  1. How to buy a car: Large capital purchases, such as buying a car or a house, can make for challenging decisions. Some input data for our car-buying determination: I have a growing family with small kids. I have a short commute to work. I care about style and comfort. I don't intend to race my car on a track. I care about the environment and would like something efficient. I have a nominal budget of $50k in mind. A cursory examination of the car market should quickly reveal a broad spectrum of options. For the sake of this post, let's narrow that down to 1) a minivan (Honda Odyssey), 2) a hybrid sedan (Prius), and 3) a pure electric (Tesla Model 3).

  2. How to pick the next feature: We make many complicated decisions at work. Product Managers and organizational leaders often need to decide what part of their product they should focus on given their goals. This strategic choice is perhaps the most impactful, on par with perfect execution. Input data: our app is in the market. It is growing slowly. Churn is higher than we'd like. Research shows that the current set of customers like the app, but don't love it. Should we focus on acquiring new users, increasing lifetime value, or churning fewer users?

How Not To Decide

1. Gut Feeling

Listening to your gut is probably the most common approach to decision making. It's the way we make most decisions - if we did an exhaustive process to decide what to eat for lunch, we'd never get anything done or be able to make any progress. Instinct is your subconscious brain pattern matching inputs with what it has seen in the past and making a quick, shortcut decision. Our brains are fantastic at taking in vast amounts of data and making gestalt decisions; don't fight your instincts.

However, when it comes to highly complex decisions, the very brain that helps us make rapid decisions and move forward with life, deludes us into making bad decisions. Our fast decision-making process is often known as the "reptilian brain" or "System 1 thinking". It is the reason we survived on the savannah: when we thought we saw a lion, our brain didn't take its time working through whether it was a bird, or a blade of grass, or a zebra. It told us to climb the tree first. Our deep-thinking, thoughtful cousins were pruned out of the family tree by the lion. These instant reactions, fight-or-flight instincts, all the shortcuts our brains use, can show up as cognitive biases in decision-making.

Let's use the car buying example. Imagine we walked into the local Toyota dealership. It's a boiling hot day in the middle of summer. Salespeople are extremely busy, overworked, and slightly rude. They give us the keys to a car that's been baking in the sun. We test drive it, hate it, and pass summary judgment: it's a pile of rubbish. The car is way too hot, takes forever to cool down, drives like a sloth. Additionally, we're unhappy about not being treated like royalty and don't want to buy from that dealer anyway — hard pass.

We have attributed the rudeness of a particular salesperson to not just the entire dealership, but all the dealers of this specific car manufacturer. This mistake is called a fundamental attribution error. We have attributed the car's inability to cool down instantly to a manufacturing flaw. We have ignored the base rate: all vehicles sitting in the sun on that day were hot and would take time to cool down. As a result of these biases, we may have discarded a perfectly reasonable option thanks to our instant decision making brain.

2. The Giant Spreadsheet

I love spreadsheets. They allow me to organize my life (and I love organization) and view things at various levels of detail. It is very tempting to distill every decision to some formula and take the flawed human out of the loop. The formula can be straightforward: weighted sums seem to do the trick. Every decision now becomes so precise, so mathematically elegant. Don't like the outcome? You must have gotten the inputs or weights wrong.

Let's visit our product feature prioritization decision. We could build features that target acquisition, LTV, or churn. Each row has a cost and an impact estimate. Any Product Manager worth their salt will come up with a table of priorities, each of these features as rows, and drop columns to show potential impact. Some complex mathematical jiu-jitsu comes next, and the potential impact column has numbers and color coding from red to green. We must pick the greenest feature because our matrix just told us so!

This approach is problematic because it reduces humans to automatons and throws out all intuition. Moreover, it overly simplifies the model by diminishing highly complex information into a number. In the made-up example above, working on notifications seems to win over everything else, using the scheme I've put in. But that the model itself is biased - cost and impact estimates might be completely bogus, our gut might tell us that focusing on acquiring new users is more important, or that the cost of doing notifications is probably higher.

Intuition is essential — our brains are pattern-matching against past experiences and predicting the future. Numerical models create a false sense of precision and delude us into trusting the models. Our minds are excellent at translating vast amounts of information into decisions, and we should trust them while finding ways to correct their shortcomings.

Decision Frameworks

The next section outlines the three decision frameworks that I have used in some shape or form. None of these frameworks are mine - I have merely adapted them for my purposes and found them to be applicable and relevant.

Framework 1: Reducing Dimensionality

The credit for this idea goes to my friend and colleague Josh Williams. The principles are easy to understand and apply on the fly, require little formal work, and help break through a decision making logjam.

Complex decisions are often challenging because they contain an overwhelming number of dimensions. Decomposing the problem results in a large number of smaller choices along each dimension. However, dimensions are not orthogonal — changes in one affect another. Trying to optimize all dimensions at the same time quickly gets overwhelming.

Take the car-buying example: we need to make individual decisions about passenger capacity, gas mileage, styling, manufacturer, safety features, cargo hauling, maintenance, buy vs. lease, and so on. In the example above, we need a car that can carry five humans, is efficient, stylish, safe, easy to maintain, and costs less than 50k. A Porsche 911 is stylish and safe, but doesn't cost less than 50k or carry five humans. A minivan fits most of the requirements but is on the lowest end of the style spectrum. A Prius is in the "meh" range on most things but does excellent on efficiency. The perfect car simply doesn't exist. What do we do?

A good approach in such circumstances is to reduce dimensionality. If you magically cared only about your budget and passenger capacity, the answer would become much more apparent. We can reduce dimensionality in 3 ways:

  1. Aggressively ignore dimensions that you don't care about. In the car example, we could stop caring about maintenance. Maintenance plans are straight forward. Almost every major manufacturer has a good policy. Let's get rid of that completely.

  2. Create "threshold" dimensions that you care about up to a certain point, but not beyond. For example, safety matters to my family, with our small children. But beyond a specific safety rating, any car is sufficiently safe, and we don't need to optimize any further.

  3. Establish trade budgets. This is not dimension reduction per se, but helpful in understanding the relationships between different things. For example, if we care more about efficiency than style, and getting a high gas mileage is worth twice as much as having a sexier car. This approach gives us a rough calculator to prioritize the dimensions we genuinely care about.

The beauty of this framework is that you can quickly sort through the dimensions that matter and devalue or completely discard the ones that don't. Moreover, when we end up with a few real choices at the end of the process, we are assured that all of them satisfy our constraints and would make us happy. Beyond this point, all decisions are good decisions.

Framework 2: Mediating Assessments Protocol (MAP)

This approach is from an excellent article by Daniel Kahneman, Dan Lovallo, and Olivier Sibony. If this summary piques your interest, I encourage you to read the article in full. It is clear, easily understandable, and practical. If you're at a tech company, such as Google or Facebook, and are using a structured interviewing process, you're already using MAP without knowing about it.

Remember the "giant spreadsheet" approach to making decisions? The problem with that approach was that it threw out all human intuition. What if we kept an element of intuition in the mix, but had a way to neutralize a variety of cognitive biases? This is the central idea behind the MAP framework proposed by Kahneman and team.

Let's revisit the feature prioritization problem. We need to make a decision on which feature to build next. There is a trap here — we can easily mislead ourselves into believing that we are following a structured process by sitting through presentations about each option, evaluated in its entirety, with pros and cons, followed by a decision making or voting meeting. This method is subject to precisely the same biases - confirmation bias for things you like, and recency bias for the last option presented. It is essentially the equivalent of a holistic gut call.

Here is the MAP alternative:

  1. Agree upfront on what the goals are. To continue with our example, let's say the objectives are 1) increase the number of daily users, 2) improve the performance of the app, and 3) reduce our operational costs.

  2. One presentation per goal. This method allows us to compare all proposals, on a particular dimension, instead of looking at all the aspects of one proposal. If we have a scoring rubric, we can score proposals per goal at this stage. These assessments are called mediating assessments.

  3. A final evaluation of all proposals, while looking at the mediating assessments. Note: we are not merely taking a weighted average of the intermediate scores. Instead, we are using our judgment at this juncture while keeping all the data in front of our eyes.

The changes seem subtle, but the impact can be profound. The best proof of this approach is in the use of a structured interviewing process to evaluate candidates. If you have interviewed at modern tech companies, like Facebook, Google, or most modern startups, you have experienced this. Instead of having each interviewer simply provide an overall score, the interview process involves a series of mediating assessments.

In structured interviewing processes, each person interviews and makes a judgment about one area of competency - coding, system design, communication, people management, etc. Interviewers score candidates per dimension. The hiring committee looks at all the intermediate scores and then determines an overall rating. This approach is different from each interviewer judging the candidate in all of the different areas and giving one overall score. Structured interviewing is the norm in almost all tech companies. Studies on personnel selection have conclusively shown that using such approaches to interviewing leads to more accurate long term outcomes.

Framework 3: WRAP

This framework is a summary of the WRAP process outlined in the Heath Brothers' fabulous book Decisive. I strongly encourage you to read the book, as well as use the summaryresources on their website (free registration required.)

The WRAP framework focuses on avoiding or overcoming cognitive biases that creep into all human thinking. It is easy to understand and practical to apply. Each maxim can be used independently toward decision making; apply a few or all.

1. Widen the frame

Let's go back to the car-buying example. We are trying to choose between a minivan or a Prius. This problem statement implies a particular frame: we have to decide between A and B.

However, the car is a means to an end - commuting to work, transporting children to school, picking up groceries, or traveling for leisure. In our choice, did we consider solving the more significant problem using some other means? Do we need a car at all? Could we use an electric bike to commute? Or Instacart for all groceries? How would that change our set of options?

A narrow frame is a common decision-making trap. It focuses our thinking on available options, instead of opening our minds to all possibilities, some of which may solve the problem in unique or non-traditional ways.

A classic sign of this trap is the "whether or not" question. When you hear your friend ask you "whether or not they should quit their job" or "whether or not they should build a feature" or "whether or not they should buy an iPad," you should smell a trap. One way out of this trap is removing the option you are leaning toward and making that a non-option. What if you absolutely could not quit your job or buy the iPad? What would you do then?

2. Reality Test Your Assumptions

When we survey the set of available options, we build models in our head of how those options are going to work out. These models get tested when they meet reality, and usually don't survive. We try to improve our models by finding evidence that supports or disproves the model. However, because of confirmation bias, we are much more likely to seek validating proof, rather than the contrary, or disconfirming evidence.

One way to get around this pernicious problem is to look for opposing or disconfirming evidence. Imagine we are in love with a particular feature. Instead of looking for reasons to support our instinct, look for the holes in our reasoning. Why could this feature fail or underperform?

How do other similar features perform? This line of questioning helps us determine the base rate. If most similar features underperform (low base rate), it is unlikely that this particular one is going to be the breakout.

Looking for disconfirming evidence can be difficult, especially when we're already heavily biased toward pursuing a particular path. One trick is to do a joint "premortem" exercise. Get together in a room, and imagine that you're six months into the future. The feature has been built and launched and isn't doing well. What went wrong?

Another approach to reality testing assumptions is to dip a toe in without diving in all the way. In the car buying example, we could rent a minivan for a week, followed by renting another car for a week, to test out what it would feel like living with that car. The cost of a mistake (perhaps you hate the way the minivan drives or turns out that the sedan is entirely too small for your family) is tiny compared to buying the car and discovering you made a mistake.

3. Attain Distance

One of the most striking passages in Andy Grove's book "Only The Paranoid Survive" is about Intel's decision to pivot from making computer memory to making microprocessors. Intel started as a memory company - and they were the world leader in manufacturing memory chips in the late 70s through the early 80s. The microprocessor business was niche, dwarfed by the massive memory business. However, the memory business was seeing enormous pressure from Japanese manufacturers and steadily losing margin. Pivoting the company from their roots, to go all-in on microprocessors was an incredibly difficult decision. Grove described how they finally did it:

— via Google Book Search

For complicated psychological reasons, we seem to make clearer decisions when we are deciding for others instead of ourselves. One of the most effective techniques for attaining distance is to ask:

"What would you tell your best friend to do in the same situation?" — Personal Context

"If you were let go and we hired someone else, what would they do in the same situation?" — Professional Context

4. Prepare To Be Wrong

We typically overestimate the impact of any particular decision. In reality, most decisions are reversible, or at least have escape hatches that are less catastrophic than we initially believe them to be. Jeff Bezos summarizes the concept of reversibility:

Some decisions are consequential and irreversible or nearly irreversible – one-way doors – and these decisions must be made methodically, carefully, slowly, with great deliberation and consultation. If you walk through and don't like what you see on the other side, you can't get back to where you were before. We can call these Type 1 decisions. But most decisions aren't like that – they are changeable, reversible – they're two-way doors. If you've made a suboptimal Type 2 decision, you don't have to live with the consequences for that long. You can reopen the door and go back through. Type 2 decisions can and should be made quickly by high judgment individuals or small groups.

— Jeff Bezos, Amazon Annual Shareholder Letter

What if we made a wrong car-buying decision? We own a car that we don't like, which we need to sell subsequently, and buy another car. There is a quantifiable dollar cost and some hassle in selling and buying cars and filing paperwork. But that's it. With that understanding, we no longer fear making a decision, knowing that the cost of reversing that decision is not life-altering.

In Conclusion

Life is full of decisions. In the majority of cases, our instinct is a great decision-maker. However, when faced with highly complex decisions, the evolutionary processes that helped us survive the lions on the savannah can mislead us into making poor, often irrational choices. Using frameworks to make such complex decisions allows us to counter some of those cognitive biases and make good long term choices.

Loading more posts…