How a forecast is scored

A forecasting tool should be judged by its record, not by what anyone says about it. So the rules for scoring a forecast are written down before a firm sees its first one, and nobody, including us, can decide afterwards what counted.

This post sets out those rules. They are also on the scoring page and in every firm’s agreement.

What a forecast commits to

A forecast is a written claim about one organisation, made on a given day, before any filing names it as the company with the problem. It always says five things:

• The company, by its registered name.
• The matter it is likely to need counsel for, and the practice that falls to.
• A timing range, in months from the day of the forecast.
• A strength, strong, moderate or early, with the number of records behind it.
• What would confirm it: the kinds of record that would settle the question.

The forecast is time-stamped when it is made. It is never edited, re-dated or withdrawn afterwards. If new records change the picture, Monition makes a new forecast and both stay on the scorecard.

Why timing is a range

A single date would be false precision. When a supplier loses its largest customer, the consequences arrive over months, not on a day. So every forecast gives a range, such as two to four months. The range is the part most likely to be wrong, which is exactly why it is scored.

We do not show likelihood percentages anywhere, for the same reason. A percentage on a single company suggests a precision nobody has. What Monition can honestly show is how many independent records point the same way, how directly, and when the matter is expected.

The four outcomes

Every forecast ends in one of four places. Only a public record can move it there: a docket entry, a registered lien or charge, a filing at a registry, a regulator notice that names the company in the forecast matter. News coverage does not count. Neither does whether the firm was instructed, which is confidential and not what is being measured.

• Confirmed: a record names the company in the forecast matter before the range closes.
• Too early: the matter happened, but the record came after the range closed. Right company, wrong timing. It is never counted as confirmed.
• Missed: the range closed and nothing appeared within a short grace period, or a later record ruled the forecast out.
• Still open: the range has not closed. Open forecasts are listed by name, but not counted either way.

There is a fifth kind of failure that a forecasting tool is tempted to leave out: the matter it never forecast at all. If the record shows a company needs counsel and Monition only notices when that record appears, the forecasting engine missed it, even though Monition still identified it. Those count as misses too.

Week 8, misses first

Every firm receives a scorecard in week 8. It opens with the misses, by company, each with the record that settled it. Then anything too early, then what was confirmed, then everything still open.

In the illustrative scorecard on our scoring page, the fictional firm has 25 forecasts at week 8. Two have been ruled out by later records: a haulier paid its sub-hauliers in full, and a revised packaging rule dropped glass from its scope. Three matters were not forecast at all; the record showed them first. Five were confirmed inside their range. Eighteen are still open, because most ranges have not closed yet. Nothing is too early, for the same reason.

At week 8 most forecasts are still open. A scorecard that hid that would be more flattering and less true.

The scorecard also splits outcomes by strength, so a firm can see whether strong forecasts are confirmed more often than early ones. If they are not, the strength labels are not earning their place, and we will need to change how they are set.

The backtest

The backtest runs Monition on an earlier stretch of the public record. It freezes the record at a cut-off date, lets Monition forecast using only what was published by then, and scores those forecasts with the same four rules against what the record showed afterwards. Company names are redacted.

The backtest is published with the misses first, as here. If the results are poor, they are published anyway, with what we changed. A tool that asks partners to act early has to show, in the open, how often early was right.