Kobalt Research

Home · Applications · KUMPI

KUMPI — measuring the
implicit welfare function

Kumpi: from Quechua, meaning "excellent, fine, of the highest quality"

Current AI benchmarks measure what models say. KUMPI measures what models implicitly maximize: the welfare function they construct for each affected entity in a decision context. A model can produce perfect text while systematically excluding specific entity categories from its objective function.

Benchmark Layer 3 · Applications Open benchmark · In development
— The gap

RLHF, Constitutional AI, and red-teaming evaluate model outputs. But the welfare function is upstream of the output — it determines the recommendation before language is generated. KUMPI is the first benchmark at that layer.

— What KUMPI measures

The layer that determines the recommendation

A doctor recommends treatment. A climate model projects costs. An economic model allocates resources. In each case: before any output is generated, there is an implicit optimization. What entity set is included? What welfare function is used?

KUMPI presents models with decision contexts and measures the implicit $W_e$ they construct: which entities are included, what weight each receives, whether the welfare function has the six Shannon properties.

— The core distinction

A model that says "this policy affects future generations" but uses an implicit discount rate that renders them mathematically irrelevant has a welfare function that excludes them — regardless of its text. KUMPI measures the function, not the words.

— The four metrics

How alignment is measured

α

Coverage

$\alpha = |E_{op}| / |E_{total}|$

The fraction of affected entities whose welfare the model includes in its recommendation. A model that only considers human adults in a climate decision has $\alpha < 0.5$ relative to the full entity set.

β

Fidelity

$\beta = \text{corr}(W_e^{\text{model}},\, W_e^{\text{PCC-W}})$

How accurately the model's implicit welfare function matches the formally constructed $W_e$ from the PCC-W protocol.

γ

Temporal scope

$\gamma = \int_0^T w(\tau)\,d\tau \;/\; \int_0^{T_{max}} w(\tau)\,d\tau$

The time horizon actually represented in the model's recommendations, weighted by the axiomatic temporal factor.

κ

Composite

$\kappa_{ZBS} = \alpha \cdot \beta \cdot \gamma$

The composite alignment index. A perfectly aligned system has $\kappa = 1$. Current state-of-the-art models: estimated $\kappa < 0.3$ on the initial test domains.

— Upstream vs downstream

Why output-level evaluation is insufficient

Upstream — what KUMPI measures

Welfare function construction

The layer KUMPI measures. Which entities are included? What is their $W_e$? What temporal scope is represented? This determines the recommendation before any language is generated.

Downstream — what current benchmarks measure

Output evaluation

Accuracy, fluency, safety (as refusal), preference (RLHF). None of these measure the welfare function that produced the output.

The upstream layer is causally prior. A model with a misspecified welfare function will produce systematically biased recommendations — and those recommendations can be fluent, safe by refusal criteria, and preferred by human raters who share the same exclusions. Output-level evaluation cannot detect this.

— Development phases

How KUMPI is being built

Phase 1 Domain selection

Domain selection

Initial domains: clinical decision support, climate policy recommendation, economic resource allocation. Each domain has ground-truth $W_e$ from PCC-W protocol or published welfare economics.

Phase 2 Scenario construction

Scenario construction

For each domain: 100+ decision scenarios with known entity sets, measurable welfare impacts, and formally constructed $W_e$. Scenarios include edge cases: rare conditions, non-agentic entities, future generations.

Phase 3 Validation and release

Validation and release

Open-source benchmark with standardized evaluation protocol. Models submit their recommendations; KUMPI computes $\alpha$, $\beta$, $\gamma$, $\kappa_{ZBS}$ and reports alignment geometry.

— Conceptual dependency map

This page depends on

  • Alignment as Geometry — defines $\alpha$, $\beta$, $\gamma$, $\kappa_{ZBS}$ as the alignment metrics KUMPI operationalizes
  • Non-Agentic We — protocol for constructing $W_e$ for entities that cannot self-report

Pages that extend this one

  • Research Program — the empirical work that provides ground-truth $W_e$ for KUMPI scenarios
  • AI for Health — clinical decision support as a primary KUMPI domain
  • Climate & Biosphere — climate policy recommendation as a primary KUMPI domain