Stay ahead of the curve
Get exclusive insights delivered directly to your inbox. Join thousands of tech leaders.
TL;DR
- Backtest the method the way you would audit a forecaster: freeze the evidence at a past date, run the method blind, then score direction, lead time and false alarms against what happened.
- Patents make the replay exact. Each one carries a publication date, so the evidence for any past year can be rebuilt without hindsight.
- Beat the baseline, not just the outcome. In all seven GetFocus backtests, filing volume pointed at the losing technology while the improvement rate pointed at the winner, years early.
- Seven cases can be cherry-picked, so ask for the aggregate: the GetFocus patent-based rate matches measured improvement rates at R² ≈ 0.7 across 150+ domains.
Backtest it. Pick a technology shift your industry has already lived through, freeze the evidence at a date years before the outcome, run the method on that evidence alone, and score what it said against what happened. Patent data makes the replay exact: every patent carries a publication date, so the evidence for any past year can be rebuilt without hindsight. GetFocus tests its method this way. Run on silver-halide film against digital imagers, it separated the winner in 1983, seventeen years before film sales went off a cliff in 2000.
What does a fair backtest of a technology forecast look like?
Forecasting researchers call it hindcasting: pretend it is a past date, forecast forward, then compare with the record. Five rules separate a real test from a flattering one.
| Rule | What it means | Why it matters |
|---|---|---|
| Freeze the inputs | The method sees only evidence published before the cut-off date | One later fact leaking in makes the whole test worthless |
| Fix the method first | Same model, same settings and same scoring for every case | A method tuned case by case is fitted to the past, not forecasting it |
| Include losers and surprises | False starts, slow burners and disruptors from outside your industry | Famous winners are easy to remember, so they are easy to "predict" |
| Replay at several dates | Read each case year by year, not from one convenient start | One lucky reading proves nothing |
| Beat a baseline | Compare with what your planning process, a trend line or an expert panel said at the same date | A method is only worth buying if it beats what you already have |
Score three things: direction (did it pick the winner), lead time (how many years before the market agreed) and false alarms (how often it backed a technology that went nowhere). Agree the pass mark before anyone sees a result, or every result will look like a pass.
Why are patents the fairest data for a backtest?
Most evidence leaks hindsight. Archived analyst reports are incomplete, news coverage was written by people who knew the ending, and an expert asked to think like it is 1995 cannot forget what happened next.
Patents carry no hindsight. Each one has a fixed publication date, so the exact set of inventions public in any year can be rebuilt. The record is also early, because technical progress is patented before it reaches the market, and it is standardised across fields and countries. GetFocus counts one row per invention, the patent family, so a company that files in fifteen countries does not read as fifteen inventions.
The GetFocus method reads the citation network of that record rather than the filing count. Two metrics, cycle time and knowledge flow, give each technology its Technology Improvement Rate. Every case in the GetFocus backtest library is computed from patents public at each date.
What should the method beat?
The hardest baseline is the one your industry already follows: where the filings and the money are going. In all seven GetFocus backtests, filing volume pointed at the losing technology while the improvement rate pointed at the winner, years early.
- Storage. In 2001 hard drives out-filed solid-state storage six to one, 3,316 patent families against 580, while solid-state improved at 49.6% a year against 29.2%. The filings crossed in 2010. In 2015 Western Digital agreed to pay about $19 billion for SanDisk.
- Lighting. In 1990, three years before the efficient blue LED existed, LED measured 32.5% a year against incandescent lighting's 8.6%. LED filings passed incandescent in 2002.
- Payments. By 1998 online payments read roughly 100% a year against cash infrastructure's 26%. Online filings passed cash infrastructure in 2015, two decades after the rate pointed there.
Backtest · Data storage · 2001–2026
The filings pointed at the loser for nine years
Hard drives out-filed solid-state six to one in 2001, while solid-state improved at 49.6% a year against 29.2%.
What the industry filed · new patent families per year
What the rate read · improvement rate per year
Computed from patents public at each date · GetFocus backtest: HDD vs SSD
A method that only echoes the filing chart adds nothing. One that reads the opposite way, and turns out right, is the reason to use it.
What does a passing result look like?
The Kodak backtest is the cleanest example, because one record holds two disruptions. It took two searches on public patent data: silver-halide film, and digital imagers.
Film improved at 23–26% a year against CCD's 30–90%. The rate separated digital from film in 1983, while film still out-filed every digital technology: it ran 2,611 new families in 1997, the year Kodak's market value peaked near $31 billion. Film sales went off a cliff in 2000, and Kodak filed for Chapter 11 on 19 January 2012.
Then the same rule called round two inside digital. CMOS passed CCD on rate in 2000, 32.3% against 30.6%, five years before CMOS out-filed CCD and fifteen before Sony ended CCD production.
Backtest · Imaging · 1980–2026
One patent record, two disruptions
The rate separated digital from film in 1983, then called CMOS over CCD in 2000, years before the filings agreed.
What the industry filed · new patent families per year
What the rate read · improvement rate per year
Computed from patents public at each date · GetFocus backtest: film vs digital
A real test also calls losers. GM paid about $50 million for rotary engine licences in 1970. From the first reading in 1972 the rotary improved at 4.8% a year against the piston engine's 6.6%, and by 2012 it was 5.5% against 13.0%. The rotary backtest flagged a dead end from the start.
Ronald S. Cok, a former Research Fellow at Eastman Kodak, said it plainly: "It would have been great if we'd had something like GetFocus to show management how serious the situation was." (Read the Kodak story.)
Which technology forecasting providers can show a real track record?
Ask every provider for the same four things: forecasts tied to dated inputs, the full list of cases including the losers, an outcome you can measure, and one aggregate statistic across many domains. Seven famous cases prove little on their own. Anyone can pick seven wins.
Forecasting platforms such as Metaculus and Good Judgment publish scored records, but they score forecasters on questions that resolve, not a method you can rerun on your own industry's past. Analyst hype cycles and trend reports are qualitative, which makes them hard to score at all.
GetFocus publishes both levels. The seven replayed cases sit in the backtest library. Underneath them, the patent-based rate matches the measured improvement rate at R² ≈ 0.7 across 150+ domains, five times the 28 domains MIT measured by hand: the forecast explains about 70% of the variation in measured rates. Independent work agrees that progress can be tested this way. Hindcasting 53 technologies, Farmer and Lafond (2016) found that forecast errors follow one consistent distribution, so every forecast can come with a known error range.
Which companies already use AI for technology forecasting, and what did they see?
Most published "AI forecasting" case studies are demand forecasting: supply chains, inventory, sales. That is a different problem. Technology forecasting asks which competing technology will win a job, and when.
On that question, the evidence comes from R&D leaders who have used it. Dr. Josef R. Wuensch, SVP R&D, Performance Materials at BASF: "The combination of smart algorithms with the individual commitment of the team from GetFocus provides essential insights for making informed strategic R&D investments and helps us innovate faster." Matteo Munari, Head of Technology at Alfa Laval, calls GetFocus "an invaluable tool for validating the insights of our in-house experts."
Can we test it on our own topic before we buy?
Yes, and you should. Pick a disruption that undeniably happened in your industry, and GetFocus shows how early the rates called it. Then the same method runs on the decision in front of you now, as a free 48-hour Decision Brief: a first-pass radar, the improvement rate of each competing technology, the monitoring triggers we would set, and a 30-minute readout with an analyst. The only input is your question; the evidence is public patent data.
This works wherever the competing technologies leave a patent trail, which covers most engineered products, materials and processes. It does not cover shifts driven by a business model or regulation alone, and where patenting on a route is thin, GetFocus flags it during scoping, before it can distort a verdict.
FAQ
What is the difference between backtesting and hindcasting?
None in practice. Both replay a forecasting method from a past date using only the evidence available then. Finance says backtesting; forecasting research says hindcasting.
How many past cases does a fair test need?
Enough to include at least one loser and one surprise from outside your industry. The rules matter more than the count: frozen inputs, one fixed method, a pass mark agreed in advance. An aggregate statistic across many domains is stronger evidence than any single case.
Can a backtest prove the next forecast will be right?
No test can. A backtest proves that the signal led the outcome, case after case, under the same rules. That is the bar any method should clear before it touches your roadmap.
Do we need to share confidential data for a backtest?
No. The GetFocus replay runs on public patent data and your question. For a sensitive decision, we walk you through an example built on your business context instead.
