A record of decisions is inert until somebody scores it. This is the session that does the scoring. Ninety minutes, once a quarter, and it is the only step in this subchapter that produces a measurable change in accuracy.
What you need
Decision records written before the outcomes were known, each containing the decision, the two or three considerations that actually drove it, a prediction specific enough to be wrong, a confidence figure, and the evidence that would have indicated you were mistaken.
Without records written in advance, the session cannot be run. Reconstructing from memory produces a pleasant hour and no information.
The session
1. Pull what resolved. Only decisions whose outcomes are now known. Anything still open goes back in the file. Expect four to ten in a normal quarter.
2. Score the prediction before reading your reasoning. Right or wrong, against the specific claim you wrote. Do this first, because reading the reasoning first will make a near-miss feel like a hit.
3. Sort each into one of four boxes. Good decision and good outcome. Good decision and bad outcome. Poor decision and good outcome. Poor decision and bad outcome. Judge the decision against what could have been known at the time, not against what you now know.
The second and third boxes are the entire point of the exercise. A good decision with a bad outcome requires no change in method and frequently gets one anyway. A poor decision with a good outcome is the dangerous case, because nothing in the organization will flag it and you will repeat the method.
4. Run the calibration tally. Group every resolved prediction by the confidence you assigned. Of everything you marked seventy percent, what share actually happened? Twenty or thirty resolved items, accumulated over several quarters, is enough to see your direction of bias. Nearly everyone has a consistent one.
5. Look for the domain pattern. Sort your misses by subject rather than by date. Errors cluster. If four of five misses involve how long something will take, or how a particular person will behave, or anything to do with a specific counterparty, you have located a domain where your instinct is not trained and can act accordingly.
6. Write one sentence. A single operative sentence carried into the next quarter. Not a list of resolutions, which will not survive week three. One sentence, specific enough to change a behaviour: I will not commit to a completion date in this category without asking the person doing the work for their own estimate first.
How it goes wrong
Doing it with your superior. The moment this session has any bearing on evaluation, it becomes an exercise in presenting well and stops producing information. Run it alone, or with one peer who has no authority over you and to whom you extend the same.
Letting it become a narrative. The pull toward constructing a story in which everything makes sense in retrospect is strong and defeats the purpose. Score first, in the boxes, then think about it.
Reviewing only the failures. Half the value sits in the third box, and nobody arrives at a review session intending to examine the things that went well.
Scaling it to the organization. The instrument works because it is private and carries no consequence. Made into a management process with a template and a submission deadline, it converts into performance theatre within two cycles.
What to expect
The first two sessions return almost nothing, because the sample is too small and the records are usually too vague. This is the point at which most people stop.
Somewhere around the fourth, with twenty-five or thirty resolved items behind you, a shape appears: a direction of bias, a domain where you are reliably wrong, a class of decision you are far better at than you assumed. That shape is worth more than anything you will read on the subject, because it is about you specifically, and there is no other way to obtain it.
Edited by Patrick J. Wolf, PhD