The Wiggum Loop: Cross-Model Evaluation and Dimensional Revision

The first time you run a research agent without an evaluator, you will probably be impressed. The output will be long. It will be fluent. It will have headings and bullet points and citations and a confident, authoritative tone that reads like something a competent analyst spent an afternoon on.

It will also, in a meaningful fraction of cases, be wrong—not in the way a hallucination is wrong, where a model fabricates a fact that is obviously false on inspection, but in the way that a first draft is wrong: structurally correct but thin in the places that matter, confident about things it should hedge, and silent about the gaps a domain expert would immediately notice. The model does not know that it does not know.

The problem compounds when you ask it to evaluate itself. In controlled experiments, outputs rated by the same model that produced them average 0.9 points higher on a six-dimension quality rubric than outputs rated by a separate evaluator model using the same rubric. The gap is largest on the Groundedness dimension—because a model that invented a claim is also the most likely to believe it is grounded.

There are two loops that address this problem. They operate at different levels and compose naturally. Understanding the distinction between them is prerequisite to understanding either one.

Two Loops: Ralph and Wiggum

The two loops compose cleanly: Ralph decides whether the task is complete; Wiggum decides whether the current iteration’s output clears the quality bar. A production pipeline typically runs Wiggum inside every non-trivial synthesis step, then returns the pass/fail result to the Ralph outer loop which decides whether to continue, revise the plan, or commit.

The Wiggum Loop in Detail

Named affectionately for Chief Clancy Wiggum of Springfield—a figure who catches most problems, misses some, and is constitutionally unable to evaluate his own blind spots—the loop has four components:

• Producer — the model that generated the output under review; never used as evaluator
• Evaluator — a separate model instance, selected deterministically from a rotation pool
• Rubric — a six-dimension scoring framework at 0.5-point resolution, 8.0 composite pass threshold
• Revision Prompt — targets only failing dimensions, not all six; focused revision produces better improvement than broad checklists

Implementation

The loop entry point in harness/wiggum.py:

def loop(
    task: str,
    output_path: str,
    models: ModelConfig,
    parent_trace: RunTrace,
) -> WiggumResult:
    content = Path(output_path).read_text(encoding="utf-8")
    round_scores: list[float] = []
    evaluator = select_evaluator(seed=parent_trace.run_id)

    for round_num in range(1, MAX_ROUNDS + 1):
        parent_trace.enter_stage(f"wiggum_round_{round_num}")
        result = _evaluate(content, task, evaluator, models)
        round_scores.append(result["score"])

        if result["score"] >= PASS_THRESHOLD:
            Path(output_path).write_text(content, encoding="utf-8")
            return WiggumResult(passed=True, final_score=result["score"],
                                round_scores=round_scores, rounds_taken=round_num)
        if round_num < MAX_ROUNDS:
            content = _revise(content, task, result, models.producer)

    Path(output_path).write_text(content, encoding="utf-8")
    return WiggumResult(passed=False, final_score=round_scores[-1],
                        round_scores=round_scores, rounds_taken=MAX_ROUNDS)

PASS_THRESHOLD = 8.0
MAX_ROUNDS     = 3

The Key Insight: Dimensions Beat Scores

The loop’s most important output is not the composite score. It is the dimensional breakdown. The evaluator scores six named dimensions independently before computing a composite:

• Relevance — does the output address the task as stated?
• Completeness — does it cover the required scope?
• Depth — is the analysis substantive rather than surface-level?
• Specificity — are claims supported with concrete detail?
• Structure — is the document organized for the intended reader?
• Groundedness — are claims traceable to retrieved sources?

Here is a representative two-round run from runs.jsonl, showing how the dimensional breakdown exposes exactly what revision must fix:

{
  "wiggum_rounds": 2,
  "wiggum_scores": [6.4, 8.3],
  "wiggum_dimensions": {
    "round_1": {
      "relevance": 8.0, "completeness": 5.5, "depth": 6.0,
      "specificity": 6.5, "structure": 7.5, "groundedness": 5.0
    },
    "round_2": {
      "relevance": 8.5, "completeness": 8.0, "depth": 8.0,
      "specificity": 8.5, "structure": 8.5, "groundedness": 8.0
    }
  }
}

The composite scores (6.4 → 8.3) tell you revision worked. The dimensional breakdown tells you what it fixed. This run had strong structure and relevance in round 1 but failed on coverage and sourcing. The revision prompt highlights only Completeness and Groundedness—not all six dimensions. Presenting the producer with a six-item checklist reliably causes superficial changes across all dimensions while missing the depth needed to fix the ones that actually failed.

Deterministic Evaluator Selection

The evaluator is never chosen arbitrarily. select_evaluator() hashes the run ID and maps it to a position in the pool:

EVALUATOR_POOL = os.getenv("HARNESS_EVALUATOR_POOL", "").split(",")

def select_evaluator(seed: str) -> str:
    if not EVALUATOR_POOL or EVALUATOR_POOL == [""]:
        return DEFAULT_EVALUATOR
    idx = int(hashlib.md5(seed.encode()).hexdigest(), 16) % len(EVALUATOR_POOL)
    return EVALUATOR_POOL[idx].strip()

The hash-based selection ensures the same run always uses the same evaluator (reproducibility), while different runs are distributed across the pool (bias dilution). A model that consistently over-scores Groundedness by 0.5 points will inflate the pass rate if used exclusively; distributed across a four-model pool, the bias becomes variance that is visible in the analytics and correctable.

The Numbers

Across 1,500 logged runs, the loop raises mean output quality from 6.87 (single-pass) to 8.12 (post-loop)—a gain of 1.25 points—and converts a 61% first-pass rate into an 89% final-pass rate. The round-by-round trajectory is consistent:

The Revision Step

Temperature during revision is 0.3 by design. Temperature 0.0 causes the producer to find the local minimum it can reach deterministically—minimal changes. Temperature 0.3 introduces enough variation for meaningful structural revision while maintaining coherence.

The revision compression mode is distinct from evaluation compression. Before evaluation, all second-level headings are preserved verbatim and bodies are condensed (the evaluator must assess structural coverage). Before revision, sections referenced in the evaluator’s feedback are preserved verbatim and everything else is condensed (the producer must see exactly what it wrote in the criticized passage).

Failure Modes of the Loop Itself

Revision regression. The producer improves a failing dimension while degrading a passing one. A document with strong Depth but weak Groundedness may emerge from revision with improved citations but condensed analytical sections. The loop’s score trajectory will show this as a non-monotonic pattern: 6.8 → 7.2 → 7.6, still below threshold after three rounds, with a document materially different from round 1 in ways that are not all improvements.

Evaluator hallucination. An evaluator that claims “the output does not address X” when X is clearly present will cause the producer to add redundant coverage of X in its revision. Most common on Completeness, where the evaluator must reason about what is absent rather than what is present.

Evaluator drift. A model’s scoring calibration shifts over Ollama version updates or quantization changes. The evaluator pool with rotation converts systematic drift into measurable variance. Periodic recalibration runs against a held-out human-rated set are the correct diagnostic.

Context starvation. When pre-evaluation compression removes too much, the evaluator scores structural completeness on a skeleton that misrepresents the content. The 6,000-character threshold was chosen conservatively on this axis.

Tuning the Loop

Tune PASS_THRESHOLD last, after establishing a baseline with the default. The right threshold depends on your task domain, quality expectations, and latency budget—not on a universal standard. The 8.0 default was calibrated for general research synthesis. Do not touch REVISION_FOCUS_THRESHOLD without A/B evidence; it is the parameter most likely to introduce subtle regressions.

What the 11% Tells You

The 11% of runs that exhaust all three rounds without passing are the most interesting part of the dataset. They are not random. They cluster around specific task types (narrow technical domains where the retrieval corpus is thin), specific failure classes (groundedness failures when claimed sources don’t exist), and specific model combinations (producers and evaluators from the same family, which share distributional biases the rubric cannot overcome).

The Wiggum Loop is not a guarantee. It is a reliable quality lift for the runs it handles, and a structured failure record for the runs it does not. Both are valuable—but only if you read the records. The run log is the data; the patterns are the explanation.