The reference class you’re building for next time
The estimate lesson told you to look at what comparable work actually took, then admitted the history probably does not exist. This is how it comes to exist — and why teams quietly refuse to write it down.
The SSO work is done. It shipped, late-ish, and everyone has moved on to the next thing. This lesson is about the twenty minutes immediately after that, which almost nobody spends and which is worth more than most of what replaces it.
Go back to the estimate lesson for a moment. It told you to build your number from what comparable work had actually taken, and then it admitted the awkward part: that history usually does not exist, so you either borrow a reference class from elsewhere or state that you have none.
It does not exist because nobody wrote it down at the end of the last project. And right now you are the last project. Whatever you record here is what the next estimate — yours or somebody else’s — gets to reason from.
What to record is much smaller than you think
The instinct is a document: a retrospective write-up with sections, lessons learned, action items. Those get written occasionally, read never, and are useless as a reference class because you cannot compare eleven prose documents.
What a reference class needs is rows. For each project, three numbers and one sentence:
- What you estimated — the range you gave, not the number people remember.
- What it actually took — measured to the same boundary you estimated to. If you estimated to first-customer-in-production, measure to that, not to code-complete.
- What consumed the difference — one sentence. Not a cause analysis; the dominant term.
For this project that is roughly: estimated 4–9 weeks, took 9, and the difference was per-provider deprovisioning behaviour plus a two-week wait on the platform-team column. One line. Eleven of those and you have something Flyvbjerg’s method can actually run on; one beautiful retrospective document and you have nothing.
Why teams do not do this
The mechanics take twenty minutes, so the mechanics are not the obstacle. Writing down we said six weeks and it took nine feels like filing evidence against yourself — and depending on where you work, that is not paranoia, it is an accurate read.
This is the information-flow problem from the escalation lesson, arriving one last time and in its most consequential form. In an organisation where a missed estimate is treated as a personal failing, people do not stop recording numbers. They record flattering ones — quietly adjusting the boundary, counting from a later start, describing the nine weeks as “including scope changes”. The data keeps arriving and it stops being true.
Where people get burned
A padded reference class is worse than no reference class. With none, the estimate lesson tells you to say so, and the reader knows how much to trust your number. With a dishonest one you get the full apparatus of the outside view — the distribution, the comparables, the confidence — wrapped around numbers that were edited to be comfortable, and it is far more persuasive than it deserves to be.
The second story
John Allspaw’s argument for blameless postmortems is usually remembered as “do not blame people”, which undersells it. The mechanism is the second story: not what someone should have done, but why the thing they did looked correct to them at the time, given what they knew (Allspaw, “Blameless PostMortems and a Just Culture”).
Applied to an estimate, the question stops being why were you wrong and becomes what made six weeks look right in week one. That is a question with a useful answer — you had no reference class, the deprovisioning risk was known but unquantified, the platform dependency had not surfaced yet — and every one of those is a thing the next project can actually do something about.
You have met this move before. It is the same one as declining to apologise for a decision that was right on the information available: judge the reasoning, not the outcome. Here it is pointed at your estimate rather than your decision, and it is what makes the recorded number safe enough to be honest.
Retrieval check
Without scrolling up: why is a padded reference class worse than having none at all? Name the specific mechanism, not just “it is inaccurate”.
Check your answer
Because of what it does to the reader’s confidence. With no reference class you are required to say so, and your estimate is correctly received as inside-view only — everyone discounts it appropriately.
A padded one delivers the entire apparatus of the outside view — comparables, a distribution, an evidenced-looking range — wrapped around numbers that were quietly edited to be comfortable. It is more persuasive than it deserves, so it produces confident wrong plans rather than appropriately uncertain ones. The padding is caused by what happens to whoever records a miss, which is why the honesty question comes before the recording question.
Using it next time
The payoff arrives on the next project, and it changes the opening move. Instead of breaking the work into tasks and summing — the inside view, which runs low by construction — you start from the rows:
“The last four integrations of this shape took 6, 9, 5 and 11 weeks. Nothing about this one looks easier than the median, and it has a dependency on another team, which was the dominant term in the two slowest. So: 8–12 weeks.”
Notice what that does in the room, which is where the delivering-the-plan lesson left you. It is not your judgment against theirs any more. It is four dated observations, and someone who wants a smaller number has to argue with the record rather than with your confidence — which is a much better conversation, and one you will usually win.
Now apply it: write the first row
Hands on
Close out the SSO project
Done when: One row exists, in a place you will find again, and you can state the second story for the miss without it being an apology.
- Decide the boundary and write it down once, at the top of wherever this lives: what counts as the start, and what counts as done. Every future row uses the same one or none of them compare.
- Write this project’s row: estimated range, actual, and one sentence on the dominant term. Resist explaining. If the sentence needs a clause defending you, it is the wrong sentence.
- Now the second story, in two sentences: what made the original estimate look right at the time? Aim for causes that were structural rather than personal — no reference class, an unquantified risk, a dependency nobody had surfaced.
- From that, name one thing you would do differently on the next project. Only one. A retro producing nine action items produces zero changes.
- Ask yourself honestly whether you would be comfortable if your skip-level read the row exactly as written. If the answer is no, do not soften the row — that is the important part. Write down that you were not comfortable, because that fact is more useful to you than the number is.
- Bring the row and the second story into the chat. I’ll check the second story for smuggled apology, which is the usual failure, and push on the boundary definition, which is the one that quietly ruins reference classes years later.
What this does not cover
The project is closed and the next one starts better. But the decision underneath it is still running — you bought rather than built, and that choice has a revisit condition attached to it that has not fired yet. When it does, you will be the person who recommended the thing, which turns out to be the hardest position to be in. That is next.
Read this next — primary source
Blameless PostMortems and a Just CultureJohn Allspaw, Etsy Code as Craft, 2012 — free, ten minutes
The post that made “blameless” standard vocabulary, and it is better than its reputation — the reputation is “do not blame people”, but the actual argument is about the second story: why an action made sense to the person taking it, with what they knew at the time. Google codified the same approach in chapter 15 of the SRE book four years later. Read it for the mechanism, not the slogan.
Stuck, curious, or think this lesson is wrong? Ask your teaching agent. The lessons are the scaffold; the conversation is where the learning gets unstuck.