Back Testing
The whole book four ways, then every method made again from each close since 2021, against what billed.
Chapter 3 of 3 · 10 min read · A guide by Broad Eye Solutions · Version 1.0 · Updated November 14, 2026
Chapters 1 and 2 built each half of the book four ways. Add them, method by method, and you have the whole book: a full forecast for the business from any of the four. This chapter adds them, then asks which to trust, and how far ahead.
The whole book, four ways
Each method’s existing book plus the same method’s net new in year.
Next year comes to $38.15M by Straight Line, $38.15M by Seasonal, $38.32M by Cohort and $38.38M by Power Law. The four land within 0.6% of each other on the total.
Close on the total doesn’t make them equally good. They differ in the months, and nothing so far says how far any of them can be trusted. A back test measures that.
How the back test works
Stand at every month-end close from December 2021 to September 2026, 58 closes in all. Make each method’s forecast for the whole book with only the statements on hand that day, then set every month of it against what the book went on to bill. Go as far ahead as each method reaches.
What You’ll Need
Nothing new: the statements you already have, and the discipline to use only what had closed at each close.
The Five Methods, As They Were Run
- Straight Line. Chapters 1 and 2. Net new: for the rest of the year, the months closed divided by last year’s share of them; next year, this year times 1.10, on a straight line from this year’s January share. The existing book: Chapter 2’s Straight Line, the same way: this year closed out by last year’s shares, next year that total grown by how much bigger the book comes in (this year’s December plus this year’s new accounts), on a straight line from this year’s January share. It reaches December of the next year.
- Seasonal. Net new: the same numbers, with every open month at its average share of the year. The existing book: Straight Line’s numbers, with every month at its average share of the year.
- Cohort. The existing book: Chapter 2’s cohort layer. Net new: an engine carries this year’s new accounts forward on the path older cohorts followed at the same age, and brings in the accounts not approved yet at the plan’s count, each billing what an account billed at the same age across the last 24 approval months. Both run on the whole book’s own time of year: each month read against the twelve around it. Past next December, the layer runs a second year from its own December, with next year’s new accounts joining in January. It reaches 24 months.
- Power Law. The same, with Chapter 2’s tier layer and each tier on its own grid in the engine. It reaches 24 months.
- Trend line. The usual first cut through the year, added here so it can be judged the same way: a straight line through the whole book’s last twelve monthly totals, carried forward. It reaches 24 months.
A forecast counts only where its method had history to read at that distance. At the December 2021 close the engine had nothing to read 24 months ahead, so that one is left out.
What The Engine Does
Net new in year for Cohort and Power Law is too many cells for a spreadsheet, so an engine runs it.
- It reads each account’s approval month, what it billed every month, and its tier.
- It assumes each cohort keeps following the path older cohorts followed at the same age, and that the whole book’s time of year repeats. Accounts not approved yet arrive at the plan’s count, each billing what an account billed at the same age across the last 24 approval months.
- It puts out each new account’s revenue for every month up to 24 months ahead, added up by approval month and by tier.
Every projection against what billed
From each close, every method’s forecast runs into months that hadn’t happened yet. Here’s how each one looked against what the book went on to bill. Switch between the closes under the title.
The strength of each method, by months ahead
Every close gives one forecast for each month ahead. Line up the misses by how far ahead they were made, and each distance gets a typical size of miss.
One month ahead, the typical miss is 3.6% for Straight Line, 1.4% for Seasonal, 0.6% for Cohort, 0.6% for Power Law and 6.2% for the trend line. At twelve months it’s 6.0%, 5.0%, 4.0%, 4.1% and 7.5%. At every distance it reaches, Straight Line misses more than Seasonal, since a line can’t see the time of year, and less than the trend line. At twenty-four months the trend line misses by 10.9%, against 2.4% for Power Law. Past twelve months the typical miss stops climbing. The forecasts that reach that far land in only three calendar years, and there the accounts already on the book run about 1.5 points of the whole book low while the plan’s count runs about 3.4 points high, so the two partly cancel.
Each method also has a band: its average miss plus or minus one standard deviation. Switch between the methods under the title. The scale stays put, so the bands can be compared.
Twelve months ahead, Straight Line’s band is about plus or minus 7.1 points, Seasonal’s 6.1, the trend line’s 9.3 and Power Law’s 4.7. At every distance Straight Line and Seasonal reach, Seasonal lands closer than Straight Line and holds a tighter band, and Cohort and Power Law land closer and tighter still. Out to two years the builds made from the accounts hold the tightest band of the five.
The year and the quarter
A budget is held to the year and reviewed by the quarter. From each planning close, here’s each method on both.
On the whole book’s year, Straight Line and Seasonal missed by 2.0% on average (they share a year), Cohort by 2.0% and Power Law by 2.1%: level. The trend line missed by 4.8%, high every year.
The quarters are where the methods part. Power Law’s quarters missed by as much as 4.3% and Seasonal’s 4.8%. Straight Line’s missed by as much as 9.0%: a straight line runs under the spring and summer and over the fall. The trend line’s first quarter ran 9% to 15% high in every year: the book bills less in January and February than in December, and the trend line can’t see it.
The split says where the year is won. On the existing book’s year the four are level, each missing by 1.3% to 1.5% on average. On net new in year, no method wins: Straight Line and Seasonal missed by 7.4% on average, Cohort and Power Law by 8.1%. Every one of them rests on the plan’s count of accounts nobody has signed yet.
Where the miss comes from
The same split, month by month: each part’s miss as points of the whole book’s month.
Twelve months ahead, Straight Line’s typical miss on net new is 1.5 points of the whole book and 4.9 on the existing book. Seasonal’s is 1.5 and 3.7, and Cohort’s 1.2 and 3.1. On the existing book each layer takes a little more off: the time of year takes Straight Line’s 4.9 to Seasonal’s 3.7, and the cohorts take it to 3.1. On net new, the straight line keeps up with Seasonal, since new accounts add up through the year, close to a straight climb, and the builds made from the accounts are closer.
2026 so far
Chapters 1 and 2 made each method’s Reforecast at the 3+9, the 6+6 and the 9+3, and stopped before what happened. Here’s what happened, through October.
On net new, from the March close to October, Cohort’s and Power Law’s forecasts ran +3.4% and +3.4% against what billed. Straight Line’s ran −19.3% and Seasonal’s −20.0%. Neither looks close on the year: both put it at $3.17M at the March close, and January to October alone went on to bill $2.68M. That leaves $0.49M for November and December, last year’s two biggest months for new accounts. On the existing book every layer stayed within about two points, Straight Line included: −1.0% from the March close, +0.2% for Seasonal.
Every method has limits
Straight Line: A Line, Not A Shape
A straight line from January’s share fits new accounts, which add up through the year. On the existing book it gets the year about right but not the months, which move with the time of year: one month ahead it misses by 3.6%, against 1.4% for Seasonal. Its band is wider than Seasonal’s and the account builds’ at every distance.
Seasonal: The Right Shape, A Wide Band
Seasonal fixes the shape. Its average miss stays within 2.0 points of zero at every distance, +2.0% at twelve months, and on the year it’s level with the account builds. But it reads the whole existing book at one rate and net new from the averages, so any one forecast can land 6.1 points either side of that, twelve months ahead.
Trend Line: No Shape, No Accounts
The trend line knows only the whole book’s last twelve totals. It can’t see the time of year, the accounts or the plan. Past a year, its band is far wider than the account builds’.
Cohort And Power Law: The Same Total
The two lines sit on top of each other out to eighteen months: 4.0% against 4.1% at twelve. At twenty-four months Power Law is closer, 2.4% against 2.5%. Splitting the book by tier barely moves the total. What the tiers add is where the year comes from: on this book, 8% of the existing book’s accounts bill 53% of its December revenue.
Both lean a little high further out: Cohort by +2.2% at twelve months and +2.0% at twenty-four, Power Law by +2.1% and +1.9%. The plan’s count adds 10% a year. On this book the new accounts that came in beat it by 5.5% in 2023, then fell short of it by 4.4% in 2024, 9.3% in 2025 and 8.6% so far in 2026.
The Engine: Only As Good As The Count It’s Fed
Every build forecasts what it can see, and the accounts already on the book can be seen. What none of them can see is the accounts not approved yet. How many, and when, is an input: here, the plan’s count. That’s why no method wins on net new’s year. Run the same engine with every account approved after the close at the latest cohorts’ pace instead, each month still to come billing what the average approval month of the last 24 billed at the same age, and it drifts low the further out it looks.
At twelve months the typical miss is 4.1% on the plan’s count and 3.4% at the latest pace. At twenty-four it’s 2.4% against 7.3%. The accounts already approved are forecast the same way in both, and at twenty-four months they run 1.5 points of the whole book low. The accounts approved after the close make the difference: 3.4 points high on the plan’s count, 5.8 points low at the latest pace. The latest pace doesn’t grow, and this book did.
The plan’s count moves the result too. Run Power Law with the plan at +5% and +15% instead of +10%:
Twelve months ahead its typical miss is 3.8% at +5% and 4.3% at +15%, both closer than Seasonal’s 5.0%. Two years out the plan decides more, 1.6% against 4.2%. On net new’s year it decides most: 6.8% at +5% and 10.7% at +15%. The method holds its edge on the months; the count you feed it sets how far out that edge reaches.
This Back Test: One Book, Overlapping Closes
Everything here is one invented payments portfolio, with sample data and no client data. The 58 closes overlap: past twelve months, each band rests on three or four calendar years of outcomes, not on 58 separate tests. Read a band as how this book’s forecasts landed, not as a promise for yours. The methods were also designed with the whole history in view, even though every forecast uses only what had closed. Run the same test on your own book before you lean on these numbers.
Here’s the band chart again.