Mar

3

Calibration of Probabilities: The state of the art to 1980
Sponsored by OFFICE OF NAVAL RESEARCH
June 1981

This paper presents a comprehensive review of the research literature on an aspect of probability assessment called "calibration." Calibration measures the validity of probability assessments. Being well-calibrated is critical for optimal decision-making and for the development of decision-aiding techniques.

The first class is calibration for events for which the outcome is discrete…. For such tasks, the following generalizations are justified by the research:

1. Weather forecasters, who typically have had several years of experience in assessing probabilities, are quite well calibrated.

2. Other experiments, using a wide variety of tasks and subjects, show that people are generally quite poorly calibrated. In particular, people act as though they can make much finer distinctions in their degree of uncertainty than is actually the case.

3. Overconfidence is found in most tasks; that is, people tend to overestimate how much they know.

4. The degree of overconfidence untutored assessors show is a function of the difficulty of the task. The more difficult the task, the greater the overconfidence.

5. Training can improve calibration only to a limited extent.

The second class of tasks is calibration for probabilities assigned to uncertain continuous quantities…. For calibration of continuous quantities, the following results summarize the research.

1. A nearly universal bias is found: assessors' probability density functions are too narrow. For example, 20 to 50% of the true values lie outside the .01 and .99 fractiles, instead of the prescribed 2%. This bias reflects overconfidence; the assessors think they know more about the uncertain quantities than they actually do know.

2. Some data from weather forecasters suggests that they are not overconfident in this task. But it is unclear whether this is due to training, experience, special instructions, or the specific uncertain quantities they deal with (e.g., tomorrow's high temperature).

3. A few studies have indicated that, with practice, people can learn to become somewhat better calibrated.

Stefan Jovanovich writes

Grant's great virtue as a warrior was that he had seen the absence of certainty in war everywhere from the front line (having a fellow junior officer lose his head to a Mexican cannonball as the two them walked forward) to the far rear (where half the sick, women and children left behind in crossing the Isthmus die from cholera because the contracted mules do not arrive; Grant saved the others by buying pack animals at the market price and then spending the rest of his tour in California and Oregon arguing with the War par5ment about the waste of funds). Man proposes and God disposes.

Gyve Bones adds:

In preparing for battle I have always found that plans are useless, but planning is indispensable.
[General Dwight D. Eisenhower]


Comments

Name

Email

Website

Speak your mind

Archives

Resources & Links

Search