A backtest asks how often a forecast came true at the probability it stated. If the 85% date held in only half the past cases, you know to leave more margin.

Open the Calibration tab #

Open a board, select Forecast, then choose Calibration. Anyone with a role on the board can read it. A Board owner creates backtests.

An origin is a past day on which a forecast is judged against what was delivered afterwards. A backtest judges many origins and counts how often the forecast held.

The Saved backtests list shows Created, Cohort, Origins, Evidence and Observed rate. Cohort says what was judged: a retrospective replay of today’s work items, or forward-captured runs that a schedule saved at the time. Evidence says how much the sample can tell. Each row says when it expires. Select a creation time to open that backtest. Observed rate names its probability, such as “62% at 85%”. Show more reads the next page, and “Showing the backtests loaded so far.” means more exist. A board keeps up to 50 backtests and runs 2 at a time; the (i) beside the list’s title says so. A reader below Board owner sees “You can read this board’s backtests but not create one. A board owner can.” in place of Create backtest.

Create a backtest #

1. Select Create backtest.
2. Under Origins, choose “Replay past origins under the current flow definition”, or “Judge a schedule’s runs that were saved at the time”.
3. Fill in the fields below.
4. Select Run backtest.

Sampling days independently carries no dependency, capacity change or seasonality. Block sampling keeps a weekly rhythm together when the deliveries batch, and falls back to single days when they do not.

A replay is not a forecast made at the time. It looks back at today’s work items under today’s flow definition, and it answers “When will this scope finish?” only. Only a schedule’s saved runs were captured at the time. Close origins share training days, and the result says so.

For example, the Checkout backtest keeps the defaults: 20 origins one day apart, an 84-day history window and 10,000 trials each. Read Observed hit rate against Nominal confidence. Well below 85% at the 85% probability means the 85% dates came too early, so leave more margin on the next forecast.

Read the result #

The strip above the result names its cohort: a retrospective replay, or forward-captured runs saved at the time. Its coverage reads “Partial” when the board held work items that Jira no longer confirmed, and the (i) says “When this backtest was computed, the board held work items that Jira no longer confirmed, and the backtest may count them.”

While the backtest runs, the tab reads “Drawing the origins”. If it judged several probabilities, Probability being tested switches between them: 50%, 70%, 85% or 95%.

Four tiles follow:

• Nominal confidence: the probability being judged.
• Observed hit rate: hits out of judged outcomes. It reads “Nothing has been judged at this probability” when none were.
• 95% Wilson interval: the sampling uncertainty, a range around that rate. It needs 10 judged outcomes.
• Evidence quality: “Insufficient observations” under 10 judged origins, “Limited sample” for 10 to 29, “30+ origins” for 30 or more. It also says whether origins overlap.

The Observed coverage meter sets the observed rate against the target. Calibration by probability lists Hits, Misses, Pending, Not known, Judged, Observed rate and 95% interval for each probability.

Historical origins lists each origin with its Training cutoff, its Target and its Outcome: “Delivered within the forecast”, “Later than the forecast”, “Not yet decided”, “Not known” or “Not judged” with the reason, such as “The flow definition changed after this origin, so it was not replayed.” or “Nothing was open at this origin.” A hit is green, a miss yellow, and an undecided or unknown outcome gray. Select Inspect origin to see how many work items were open at that origin, when they were delivered, or “No completion observed inside the horizon”, and when its window ends. For a saved run, select Inspect run, or Open the schedule’s run history.

For “How much can finish by a date?”, the Target reads “At least 12 by” and a date, for example. Inspect origin then counts the work items delivered inside the window:

• “At least 9 work items inside the window”: the count is a floor, since some counted work items, or ones that left the board, may have delivered unseen.
• “3 more work items may have delivered inside the window”: no kept evidence dates their first delivery inside or outside the window.
• “2 work items left this board with no delivery on record”: they left before a delivery was kept, so a level that turns on them reads “Not known”.
• “A backtest reads at most … that left this board. The later days were not read, so more may have delivered inside this window.”: the count stops at that limit, so read it as a floor.

The card Coverage: the origins this backtest could not judge counts the judged, undecided, not known and left-out origins, with a reason for each group. Left-out origins stay outside the hit rate. Each reason is a limit of the backtest, never a hit and never a miss:

What this backtest was computed over gives the period, horizon, trials and Kept until. What this backtest cannot tell you lists the limits that hold for this result, such as “Retrospective replay of the current cohort, not forecasts captured at the time”, “Experimental: the observed history behind this is thin” or “Deliveries batch on release days, and the model samples days”.

On the Standard edition #

Saved backtests stay readable, each marked with the yellow lozenge Retained result: produced under the Advanced edition. The tab says “New backtests pause on this edition. What is here stays readable.” With none saved, it shows “This is part of the Advanced edition”.

If something looks off #

• “No backtests yet”: no backtest was created on this board. Create one to see how often each stated probability came true.
• “The board’s configuration changed after the data replayed here was published; …”, “A refresh of the board failed after the data replayed here was published, …” or “The data replayed here was calculated before a correction of …”: this result may not match the board’s data now. Create the backtest again after the board’s next data update.
• “Not shown below 10 judged outcomes”: add origins so the interval has enough to stand on.
• “This backtest cannot be run yet”: the dialog lists what to fix, such as a total above 6,000,000 trials.
• “This board has no schedule with retained runs.”: create a schedule and let it run first.
• “This backtest is paused.”: its creator selects Resume on the backtest’s job line; only people other than the creator see this message.
• “This backtest did not finish”: create it again.
• “Backtest expired”: create a new one.