• Modern Generative Models: Broad capabilities & vulnerabilities; require iterative checks for safety and performance

• Challenge: Comprehensive testing is resource-intensive

• Current Common Practice: Estimate performance on a subset of test bank using average scores via Classical Test Theory (CTT)

• Limitations: Average scores are only compatible on parallel tests, which have similar difficulty levels

• Challenge of CTT: Parallel test is often hard to construct (e.g., web-based environments, healthcare, adversarial red-teaming)

• Example: Evaluating LLMs’ agentic capability on the web. Due to pure environment stochasticity, two agents with identical ability might experience two distinct trajectories with varying difficulty, hence yield different average score.

• Thesis: In CTT-based generative model evaluation, random subset selection improves efficiency but reduces reliability (test-invariant ability estimation). Can we achieve both?

• Contribution 1: To address test-dependent ability estimation of CTT, let’s explicitly decouple item difficulty and test taker ability via an explicit probabilistic model, via Item Response Theory (IRT).

• Contribution 2: IRT requires calibration of item parameters, which is often expensive to obtain. We propose amortized calibration to reduce the cost complexity from linear to constaint.

• Contribution 3: IRT improves evaluation efficiency beyond random subset via adaptive item selection, but relies on a diverse item bank, which is costly to construct. We overcome this with conditional item generation.

• Section 2: Background: Rasch Model and Fitting
• Section 3: Generalization of Estimated Ability on Random Subset
• Section 4: Amortized Calibration
• Section 5: Adaptive Test with Conditional Item Generation

• A test giver interacts with a test taker whose fixed, unknown ability $\theta$ is sampled from population $p(\theta )$.

• An item $q$ is generated conditioned on a latent variable $z$ sampled from a latent distribution $p(z)$.

• A Bernoulli random variable $y$ indicates whether the test taker answers the question correctly, with $y=1$ for a correct answer and $y=0$ for an incorrect one.

• In IRT, the Rasch model describes the probability of a correct answer as a function of the test taker ability $\theta$ and the item difficulty $z$, using the logit $\sigma$ function:

  $$p(y=1\mid \theta ,z)=\sigma (\theta -z).$$

• A measurement using IRT includes two phases: (1) calibration and (2) adaptive testing.

• In the calibration phase, we collect a response matrix, denoted as $Y\in {\mathbb{R}}^{M\times N}$, where $M$ denotes the total number of test takers, and $N$ denotes the total number of items.

• Each binary entry $Y_{i,j}$ represents the response of test taker $i$ (with ability ${\theta }_i$) to item $j$ (with difficulty $z_j$).

• With the response matrix, ability and difficulty parameters can be estimated via e.g., Expectation Maximization (EM) algorithm.

• EM estimates the difficulty parameters $\{z_j\mid j\in [1,N]\}$ by alternating between:

$$\text{E step:}p(Y_{i,j}|z_j^t)={\int }_{{\theta }_i}p(Y_{i,j}|{\theta }_i,z_j^t)p({\theta }_i)d{\theta }_i$$

$$\text{M step:}{z}_j^{t+1}=\arg \underset{z_j^t}{max}\sum _{i=1}^M\log p(Y_{i,j}|z_j^t),$$

• With estimated difficulties, we can estimate the abilities of the test takers ${\theta }_i\mid i\in [1,M]$ using e.g., MLE:

$$\widehat{{\theta }_i}=\arg \underset{{\theta }_i}{max}\sum _{j=1}^N\log p(Y_{i,j}|{\theta }_i,\widehat{z_j}).$$

Our empirical analysis is conducted on 25 datasets from HELM (involving both capability and safety datasets) and 184 large language models. Responses are dichotomously scored (correct: 1, incorrect: 0)

• To validate model-based evaluation with IRT, we conduct a case study assessing test takers using dataset subsets. We randomly select a test taker $i^{\ast }$, estimate their ability ${\theta }_{i^{\ast }}$ on one subset, and test its generalizability on another.

• All other test takers serve as side information. Two disjoint subsets of 50 items are randomly sampled—one for estimation and the other for evaluation.

• CTT estimates correctness by averaging scores from the first subset and applying them uniformly to the second, while IRT estimates $\theta$ using Rasch’s model and item parameters $z$ from other test takers to predict correctness.

• Model performance is evaluated using AUC-ROC on the second subset, repeated across 10 test takers with 10 subset pairs each. IRT achieves an average AUC-ROC of $0.78\pm 0.07$, whereas CTT performs at chance level ($0.5\pm 0.07$).

• Recalibration is essential for periodic item bank updates.

• Challenges: Calibration costs scale linearly with the number of items.

• Amortized calibration: Predicts item parameters directly, eliminating the need for collecting responses for each new item, reducing calibration costs to constant.

• Amortized calibrations is effective across multiple datasets, including unseen ones, without prior calibration.

• In the adaptive testing phase, the goal is to reliably estimate the ability of a new test taker ${\theta }_{\text{new}}$, using only $K$ items, where $K<<N$.

• With the difficulty information, the test giver can select the next item based on the test taker’s current estimated ability in each iteration.

• The item selection is guided by an acquisition function e.g., Fisher information, iterating between:

$$q^{\ast t}=\arg \underset{q_j\in {\mathcal{Q}}^t}{max}\mathbb{I}({\theta }_{\text{new}}^t;\widehat{z_j}){\mathcal{Q}}^{t+1}={\mathcal{Q}}^t\setminus \{q^{\ast t}\}$$

$${\theta }_{\text{new}}^{t+1}=\arg \underset{{\theta }_{\text{new}}^t}{max}\sum _{j=1}^t\log p(Y_{\text{new},j}|{\theta }_{\text{new}}^t,\widehat{z_j}),$$

where $t\in [1,K]$ is the iteration index, the set $\{q^{\ast t}\mid t\in [1,K]\}$ is the final selected item set of size $K$, and $\mathbb{I}({\theta }_{\text{new}}^t;\widehat{z_j})=p(1-p)$ is the Fisher information, with $p=p(Y_{\text{new},j}|{\theta }_{\text{new}}^t,\widehat{z_j})$.