How 24 questions can pinpoint a learner — and why static tests don’t.

Walk into a typical corporate L&D rollout. Every new hire takes the same 100-question placement test. The strong engineer is bored for the first 60 questions, which are too easy. The new-to-the-domain hire is overwhelmed by the last 40, which are way too hard. After two hours, the system places each of them — but the test that determined their starting point delivered almost no useful information about either.

This is the static-test problem. The fix has existed since the 1950s but is still rare in enterprise learning platforms: an adaptive test that figures out who you are while you’re taking it, and uses each question to learn more about you than the last.

The starting point: Lord, 1952

Frederic Lord’s 1952 monograph [1] introduced what became Item Response Theory. The core insight: rather than treating a test as a sum of items, treat each item as a probabilistic function of the test-taker’s underlying ability.

Concretely: for each test item, there is a probability of correct response that varies with the test-taker’s ability. For an easy item, almost everyone gets it right, with the probability rising sharply around a low ability level. For a hard item, almost nobody gets it right at low ability, but the probability climbs steeply once ability exceeds a threshold. Plotting this probability against ability gives the Item Characteristic Curve (ICC) — a logistic curve characterized by its difficulty (b-parameter) and its discrimination (a-parameter).

Once you have an item bank with calibrated ICCs, you can run the test as follows: start with an item of average difficulty. Observe whether the learner gets it right. Update your estimate of their ability (Bayesian or Maximum Likelihood — both work). Pick the next item to maximize information about ability at the current estimate. Repeat until the precision crosses a threshold.

This is Computerized Adaptive Testing, and it’s been the standard in high-stakes psychometrics since the 1980s. The GRE, GMAT, and every modern licensure exam use it. Most learning platforms, somehow, still don’t.

Why 24 items beats 100

The number “24” isn’t arbitrary. It comes from the math of Fisher information.

Each item answer reduces uncertainty in your ability estimate by an amount that depends on how well-matched the item was to your current estimate. A perfectly matched item (its difficulty equal to your ability) contributes maximally; an item that’s way too easy or way too hard contributes nearly nothing — you got it right (or wrong), but you would have gotten it right (or wrong) at most ability levels.

In a 100-item static test, perhaps 15–20 items happen to be at your ability level. The other 80 carry almost no information about you. They produce noise — and frustration. The same 15–20 items, selected adaptively, are the only ones a CAT would have given you in the first place.

Practically, this means CAT can deliver:

• Equivalent precision in ~25% of the time.
• Adjustable precision — stop early for rough placement, run longer for high-stakes.
• A reasonable test experience: you’re never bored, never crushed; almost every question feels “right at the edge.”

Going beyond 1D ability: Bloom-aware CAT

Classical IRT assumes a single ability dimension (θ) per content area. This is a useful simplification for math placement or vocabulary tests — but it falls apart when you ask the deeper question that L&D leaders actually care about: at what level of Bloom’s taxonomy [7] is this learner functioning for this concept?

A learner might score well on “Remember” items (define a function) but fail on “Apply” items (write a function for a novel problem). One-dimensional CAT averages these and reports a single ability — which loses the most important signal in the data.

Future Proof’s diagnostic runs a separate CAT per Bloom level, with shared priors across levels (a strong “Remember” score raises the prior on “Understand” — there’s a partial-credit Bayesian network running underneath). The result: not a single placement score, but a Bloom profile per concept. This is what makes the “weak on Apply” classroom view (visible to teachers and L&D admins) possible.

What it takes to do this well

Most of the difficulty in real-world CAT isn’t in the math — the math is settled — but in the operational disciplines around the item bank:

• Item calibration. Each item’s a and b parameters must be estimated from a sufficient sample (typically 200–500 responses per item) before it’s safe to use in adaptive selection.
• Item exposure control. If one item is “optimal” for many learners at the same ability, it gets shown too often and leaks. We use Sympson-Hetter or similar randomization to spread exposure.
• Calibration drift. Item parameters change over time as content meaning shifts (e.g., a question about cloud computing is different difficulty in 2010 vs. 2024). We re-calibrate quarterly.
• Item bank size. CAT needs many items per ability range. Generating these manually is a bottleneck — which is why the AI Question Studio exists.

What we don’t yet know

Three questions our research team is actively studying:

• How do item parameters drift across language editions? The same item, translated to Hindi, doesn’t necessarily preserve its a and b parameters. We’re building cross-language calibration models with a state education department.
• Can LLM-generated items match human-authored items in IRT fit? Empirically, LLM-generated items have higher variance in discrimination — some are excellent, some are noise. Our QC pipeline is built to detect and prune the noise, but the underlying question is open.
• What’s the right CAT termination criterion in a learning (not assessment) context? For grading, a fixed standard error works. For diagnostic placement before adaptive practice, the answer is less clear — we’re testing several rules.