Two people sit on the same board. One is certain about everything and turns out to be right rather less often than his manner suggests. The other qualifies every statement, declines to commit, and is described admiringly as measured.
They have the same defect. Neither can distinguish between the questions they understand well and the questions they do not, and the second one has simply chosen a failure mode that attracts compliments.
Calibration is not humility
Humility is a disposition. Calibration is an accuracy property, and the difference matters because one of them can be measured.
A well calibrated person is one whose confidence tracks their hit rate. When they say they are seventy percent sure, they are right about seventy percent of the time. When they say ninety, they are right about ninety. This says nothing about whether they are generally confident or generally cautious. A person can be calibrated at high confidence, which is what genuine expertise looks like, or calibrated at low confidence, which is what honest inexperience looks like. Both are useful to work with. What is not useful is a person whose stated confidence bears no relationship to anything, and that description covers most people most of the time.
The chronic hedger deserves particular attention, because organizations reward the behaviour and rarely notice the cost. Refusing to commit is not caution. It is a transfer: the person retains the appearance of prudence while pushing the actual decision onto someone else, and if it goes badly they were on record as uncertain. An organization full of measured people who never state a probability cannot learn anything, because there is nothing on the record to check.
Words will not carry this
The instinct is to express uncertainty in language. Likely. Probably. A real possibility. Some chance.
Sherman Kent, running analysis at the CIA in the early 1960s, ran into this directly. An estimate had used the phrase “serious possibility” about a Soviet action, and Kent, curious, asked colleagues what odds they had understood by it. The answers ranged from roughly one in five to something close to even. The people writing and the people reading had believed they were communicating. Kent’s response was to push for numbers, on the grounds that a phrase which means twenty percent to one reader and fifty to another is not conveying an estimate at all.
The same collapse happens in every board room in the state. A chief executive says a project will probably come in on budget, three directors hear three different things, and nobody discovers the gap until the variance report arrives. The correction is unglamorous and effective: attach a number. Not because the number is precise, but because it can be wrong in a way that a word cannot.
Where your confidence is systematically wrong
Miscalibration is not distributed evenly across your life. It concentrates in predictable places, and knowing the pattern is most of the remedy.
Confidence runs highest, relative to accuracy, in the areas where you are praised. Reputation is a form of feedback that is entirely uncorrelated with being right, and it accumulates faster than correction does. It runs high in areas where you are the senior person present, because nobody in the room is positioned to tell you the estimate is off. It runs high in anything involving people and time, both of which almost everyone is confidently wrong about in the same direction. And it runs high in domains adjacent to your real expertise, which is the most dangerous case, since the adjacent domain feels familiar and supplies none of the actual repetitions.
Confidence runs lowest, relative to accuracy, in areas where you were once badly wrong in public. A single humiliating error will suppress a person’s stated confidence in a whole domain for a decade, well past the point where the underlying competence has recovered.
What it costs the organization
A leader whose confidence is uncalibrated teaches everyone around them to apply a discount, and the discount is applied indiscriminately.
People work out fairly quickly that this person always sounds certain, and they begin subtracting a fixed amount from everything the person says. The subtraction is then applied to the statements that were accurate as well, including the urgent one, including the warning that mattered. The leader experiences this as an organization that does not listen. What has actually happened is that they spent their signal on things they did not know, and it was not available when they needed it.
The reverse is equally real. A leader known to be calibrated can move an organization with a sentence, because when they say they are quite sure, everyone understands that to be information rather than temperament.
Measuring it
Calibration is one of the few things in this subject that can be scored, and the exercise takes very little effort for how much it returns.
Attach a percentage to perhaps ten predictions a quarter. Only ones that will resolve, and resolve within a period you will still care about. Then, when they resolve, group them: of everything you called seventy percent, how many happened? Twenty or thirty resolved predictions is enough to see the shape of your own bias, and nearly everyone discovers a consistent direction rather than random noise.
The goal is not to become less confident. Someone who responds to this by hedging everything has made themselves useless in a new way. The goal is that your stated confidence should carry information, so that the people who depend on your judgment know when to plan around it and when to go and check.
Edited by Patrick J. Wolf, PhD