Using Analytics for Decision Making
Executive Readiness Lens
Analytics earns its keep only when it changes a decision. The senior manager's job is not to build the model; it is to frame the decision question sharply, define acceptable error, and ensure the output closes the loop into action. In practice, that means deciding when evidence is good enough, when a pilot is needed, when the simplest model is the right model, and when an intuition override is justified.
Canonical Grounding
- Simon: managers satisfice under bounded rationality, so analytics should reduce decision uncertainty, not create endless search.
- Kahneman: analytics supports slower, higher-quality judgment where intuition is vulnerable to bias.
- Hubbard: measure the uncertainty that matters most; do not collect data for its own sake.
- Decision science principle: the question comes before the model.
Working Heuristic
Use this five-step bridge to convert analytics output into better decisions:
- Decision first. State the exact decision, the decision owner, and the deadline before any data pull.
- Error cost next. Decide whether false positives or false negatives are more expensive.
- Pilot when history is thin. Generate the data through a controlled experiment instead of waiting for perfect history.
- Simplest model wins unless proven otherwise. A simpler model that beats the baseline is usually preferable to a fragile complex one.
- Close the action loop. Analytics output must change a named decision on a named cadence.
When to use this framework
- Before any new analytics project.
- When deciding whether a prediction is worth the cost.
- When reviewing whether existing models are actually used.
Key leadership principle
The most valuable analytics skill is asking the right question, not building the right model.
Corporate Reality Check
Run this reality check before reviewing analytics-to-decision conversion quality in your team or function. This surfaces decision theatre — analytics output that exists but changes no decisions.
Common failure patterns:
- Decision questions added after model design rather than before it.
- No named owner for acting on model output within a defined cadence.
- Pilot studies designed without pre-defined success metrics and decision rules.
Failure signals:
- Analytics team growing while business decision quality stays unchanged.
- Models deployed but recommendations are not incorporated into operating decisions.
- Senior leaders override analytical recommendations without documented rationale.
What to do instead: Require every analytics project to pass a three-question gate before starting: What decision changes? What is the cost of being wrong? What is the simplest model that beats current approach?
Case Lens
Billy Beane’s Oakland A’s show the power of reframing the decision question. Instead of asking "who is the best player?", they asked "which player produces the most wins per dollar spent?" That shift surfaced undervalued players that conventional scouts ignored and allowed a low-budget team to compete with far richer rivals.
Lesson: reframing the question is often more valuable than building a better model.
Full Case Walkthrough
Click here to read full case study
1. Case Context
Major League Baseball in the late 1990s relied heavily on scout judgment and visible metrics like batting average and home runs. The Oakland A’s faced a major payroll disadvantage, so they had to rethink how player value was measured.
2. Decision Trigger
Three conditions forced the pivot: the A’s could not compete on payroll, the market was mispricing traditional metrics, and the team had the quantitative capability to use sabermetrics.
3. Timeline (Simplified)
| Phase | What Happened | Management Relevance |
|---|---|---|
| Conventional approach | Scout-led evaluation using traditional metrics | Player pricing reflected conventional wisdom |
| Decision question reframe | Shift from "best player" to "most wins per dollar" | On-base percentage became the key metric |
| Model build and acquisition | Statistical models identified undervalued players | The A’s bought wins at a discount |
| 2002 season | 20-game winning streak; competitive with teams at 3× payroll | Analytics advantage showed up in results |
4. Options Considered (Managerial Framing)
Three options existed: stay within the conventional scouting frame, focus only on player development, or reframe player value and exploit market mispricing. The A’s chose the third path.
5. Execution Moves
The key execution moves were: redefining the decision metric, using simple models, buying undervalued players, and resisting pressure to revert to traditional scouting.
6. Outcomes and Evidence
What improved: the A’s competed effectively against richer teams. What remained challenging: post-season variance and later imitation by competitors.
7. What to Transfer to Managerial Practice
What to copy:
- Reframe decision questions around the variable that most directly predicts the outcome you care about.
- Start with simple models that beat the current baseline.
- Build discipline against organisational pressure to revert to conventional wisdom.
What to adapt:
- The specific metrics that predict outcomes vary by domain and market.
- As analytical methods spread, advantage shifts to better data or faster iteration.
What to avoid:
- Assuming the most sophisticated model is always the best model.
- Ignoring the cost of analytical error types.
8. What We Know vs What Is Inferred
| Category | Statement Type |
|---|---|
| What we know (documented) | Michael Lewis documented the Oakland A's 2002 season and analytical methods in Moneyball. |
| What is inferred (managerial synthesis) | Information-based competitive advantage decays as competitors adopt similar methods. |
9. Discussion Questions
- What is your equivalent of on-base percentage?
- Where would a pilot study beat a predictive model in your business?
- How would you explain false positive versus false negative cost to a non-quantitative stakeholder?
- When should intuition override analytics, and how should that be documented?
- What would change if every analytics project had to pass the three-question gate?
Monday Morning Playbook
30-minute prep
- List the three most important decisions your team will make this month and map each to its current evidence base.
- Identify one decision currently made on gut feel where a simple model could help.
- Review one active analytics project and confirm the team can state the decision it will change.
60-minute decision analytics review
- Review the decision quality for the top three operating decisions from the past week.
- Evaluate one analytics project for decision-question clarity and simplest-model discipline.
- Confirm that one pilot study has pre-defined success metrics, control group, and decision rule.
- Document one analytical recommendation that was overridden and why.
7-day follow-through
- Apply the three-question gate to every new analytics request this week.
- Ask your analytics team to name the decision changed by each of the top five active models.
- Calculate the cost of your most common analytical error type in one key process and set the next error threshold accordingly.
Role-Based Activation
- People Manager: require your team to state the decision question before analysis starts.
- Functional Leader: audit analytics projects for decision linkage and cut weak ones.
- BU Leader: set an explicit decision-quality improvement target in the annual plan.
- Strategy/Founder Office: review the analytics portfolio against the three-question gate.
KPI and Evidence Block
Track leading, lagging, and risk indicators in one evidence view.
| Metric Type | Suggested Metric | Review Cadence |
|---|---|---|
| Leading | % active analytics projects with decision question, owner, and error cost defined | Monthly |
| Leading | % pilot studies with pre-defined success metrics and decision rules | Monthly |
| Lagging | Decision accuracy improvement vs baseline for analytically supported decisions | Quarterly |
| Lagging | % major decisions made with analytical input vs gut feel | Quarterly |
| Risk | % analytics outputs with no linked recurrent decision or named action owner | Monthly |
Tools Pack
Tool 1: Three-Question Analytics Gate
Apply before approving any analytics project. All three must be answered.
Question 1: What specific decision will this output change?
Answer: _______________________
Question 2: What is the cost of being wrong? (false positive vs false negative)
False positive cost: _______________________
False negative cost: _______________________
Question 3: What is the simplest model that beats the current approach?
Current baseline: _______________________
Minimum viable model: _______________________
Tool 2: Pilot Study Design Template
Use when no historical data exists for the decision at hand.
| Element | Your Answer |
|---|---|
| Decision to be made | |
| Treatment group (who gets new approach) | |
| Control group (who continues with current approach) | |
| Success metric (pre-defined) | |
| Decision rule (what level triggers the decision) | |
| Duration | |
| Sample size | |
| Who reviews and decides |
Practice MCQs
A consumer goods firm is launching a new flavour with no prior sales history. What is the most appropriate analytics approach?
- A. Build an ARIMA time-series model on 5 years of data
- B. Run a 6-week pilot in 3 cities using matched control markets
- C. Deploy a neural network on social media data
- D. Wait 12 months for sufficient data
What is 'decision theatre' in analytics?
- A. A dashboard displayed in a board meeting
- B. Analytics output that is produced but does not change any actual decision
- C. A theatre performance about data science
- D. Real-time analytics displayed on large screens
In a fraud detection model, which error type is most costly?
- A. False positive (flagging a legitimate transaction as fraud)
- B. False negative (missing a fraudulent transaction)
- C. Both are equally costly
- D. Neither is costly if the model has 95% accuracy
A manager receives a churn prediction model output showing 10,000 customers 'likely to churn.' What is the most important next question?
- A. How was the model trained?
- B. What action will we take differently for these customers vs our default process, and what is the expected ROI?
- C. What is the model’s accuracy score?
- D. Who built the model?
Seasonal products need time-series models trained on at least:
- A. 1 week of data
- B. 1 month of data
- C. 2–3 full seasonal cycles (typically 2+ years)
- D. Any amount — larger is always better
Why should a manager ask 'What is the simplest model that beats our current approach?' before commissioning analytics?
- A. To reduce the data science team’s workload
- B. Because complex models are never useful
- C. To prevent over-engineering — the marginal gain of a complex model often does not justify its cost, maintenance, and explainability burden
- D. To comply with GDPR
The 'build vs buy' analytics decision should favour building internal capability when:
- A. Analytics is a commodity support function like payroll processing
- B. Analytics is a core competitive differentiator tied to proprietary data
- C. The team is small and data is scarce
- D. The vendor solution is already available
Which of the following is a well-framed analytics decision question?
- A. Analyse our customer base
- B. Should we offer a 20% retention discount to customers in the high-value/high-churn-risk segment in Q4, and what is the predicted ROI?
- C. Improve our data quality
- D. Build a machine learning model
What is the primary risk of using a very large historical dataset for predictive modelling?
- A. Data storage cost
- B. Older data may reflect outdated patterns that no longer predict current behaviour (staleness)
- C. Models always perform better with more data
- D. GDPR compliance issues only
A management consultant's rule of thumb for minimum sample size in cross-sectional regression models is:
- A. At least 5 observations
- B. At least 30 observations per explanatory variable
- C. 10,000 observations regardless of variables
- D. No minimum — more is always better
Flashcards
Click the card to flip
Continue Learning
References
- Hubbard, D. W. (2010). How to measure anything (2nd ed.). Wiley.
- Kahneman, D. (2011). Thinking, fast and slow. Farrar, Straus and Giroux.
- Lewis, M. (2003). Moneyball: The art of winning an unfair game. W. W. Norton & Company.
- Simon, H. A. (1955). A behavioral model of rational choice. Quarterly Journal of Economics, 69(1), 99-118. https://doi.org/10.2307/1884852
The most important analytics skill is asking the right question, not building the right model.