Harness vs. Perplexity: Eight Iterations to Parity

Previous experiments in this series used a controlled within-pipeline design: same model, same evaluator, same research context, with one variable changed at a time. The perplexity_vs_harness experiment is different. It uses a frozen external output — a real response from Perplexity AI — as the comparison condition, evaluated through the same Wiggum scorer as every harness output. The isolation is tight: same task string, same evaluate() call, document source is the only variable.

The specific task was chosen to favor the harness: current-conditions Fed district inflation analysis as of early 2025, where live FRED series and fresh Beige Book retrieval should provide genuine information advantage. The initial assumption was that Perplexity would be working from training data with a recency ceiling. That assumption turned out to be wrong, and it changes what the experiment actually measured.

Experiment Design

The task asks about a domain where the harness has a designed advantage — live quantitative data, district-level qualitative retrieval, and structured synthesis instructions all exist specifically to answer questions like this one:

The Perplexity output was captured from a live session, copied verbatim, and stored as a frozen constant in the run script. This ensures the perplexity condition never changes across re-runs, and that any delta movement is attributable to changes on the harness side.

The Eight Iterations

The experiment ran eight times across two phases. The first four fixed infrastructure failures; the next four chased the depth score. Here is what each one found.

Bug Deep-Dives

FRED HTTP 429 — Rate Limiting Architecture

The FRED API allows approximately 2 requests per second. The original fred_tool.py added a _SERIES_DELAY = 0.6s sleep between series calls, but get_series() makes two _get() calls internally (one for metadata, one for observations). With 5 series and 2 calls each, that's 10 HTTP requests with only 4 inter-series sleeps — bursts that reliably exceeded the rate limit.

The fix moved the throttle from between-series to per-request, using a module-level timestamp:

_MIN_INTERVAL = 0.7  # ~1.4 req/s — safely under FRED's 2 req/s limit
_last_req_t: float = 0.0

def _get(endpoint, params):
    global _last_req_t
    elapsed = time.monotonic() - _last_req_t
    if elapsed < _MIN_INTERVAL:
        time.sleep(_MIN_INTERVAL - elapsed)
    for attempt in range(_MAX_RETRIES):
        try:
            with urllib.request.urlopen(url, timeout=_TIMEOUT) as resp:
                result = json.loads(resp.read().decode("utf-8"))
            _last_req_t = time.monotonic()
            return result
        except urllib.error.HTTPError as e:
            if e.code == 429 and attempt < _MAX_RETRIES - 1:
                wait = _RETRY_BASE * (2 ** attempt)
                time.sleep(wait)
                _last_req_t = time.monotonic()
                continue
            raise

With _MIN_INTERVAL = 0.7s, every individual HTTP call is rate-throttled regardless of which function calls it. The inter-series sleep was removed entirely since the per-request throttle makes it redundant. _MAX_RETRIES = 4 and exponential backoff starting at 3s handle transient overload.

BEA 'list' object has no attribute 'get'

The BEA API returns BEAAPI.Results as either a dict or a list depending on the dataset. GDPbyIndustry returns a list. It also nests data inside results["GDPbyIndustry"][n]["Data"] rather than the flat results["Data"] path that all other datasets use. Two code paths needed fixing:

# Fix 1: normalize list Results at parse time
results = raw.get("BEAAPI", {}).get("Results", {})
if isinstance(results, list):
    results = results[0] if results else {}

# Fix 2: GDPbyIndustry-specific data extraction
if dataset == "GDPbyIndustry":
    gdpi_list = results.get("GDPbyIndustry", [])
    data = []
    for item in (gdpi_list if isinstance(gdpi_list, list) else []):
        if isinstance(item, dict):
            data.extend(item.get("Data", []))
    if not data:
        data = results.get("Data", [])
else:
    data = results.get("Data", [])

Alpaca False Positive on "signal"

The task string contains the phrase "what the latest Beige Book reports signal about near-term conditions." The original _has_trading_intent() function used a token-level set membership check. The word "signal" was in _TRADING_KEYWORDS. So was "short," "long," "position," and several other words that appear naturally in economic analysis prose.

When Alpaca fired, it injected a prompt preamble about the user's paper trading portfolio (TSLA, GOOG, NVDA holdings, stop-loss levels, position sizing rules) into the front of the research context. The synthesis model then spent part of its output addressing portfolio implications of Fed district inflation — a coherent but completely off-task tangent that hurt depth and specificity on the actual task.

The fix replaced token matching with phrase matching, requiring unambiguous multi-word trading phrases that cannot appear in economic research prose:

_TRADING_PHRASES = frozenset({
    "paper trade", "paper trading", "alpaca",
    "buy signal", "sell signal", "long thesis", "short thesis",
    "trading thesis", "trade thesis", "entry point", "exit point",
    "position size", "portfolio allocation", "place order", "limit order",
    "market order", "trade idea", "trade setup", "trade recommendation",
    "actionable trade", "open position", "long position", "short position",
})

def _has_trading_intent(query: str) -> bool:
    q = query.lower()
    return any(phrase in q for phrase in _TRADING_PHRASES)

Evaluator Rubric Mismatch

This was the most subtle failure mode. The Wiggum evaluator's EVAL_PROMPT defines depth partly as: "Does the response include implementation notes, named tools, worked examples with parameters?" That language is appropriate for a software engineering best-practices task. It is not appropriate for an economic research synthesis.

TASK_CRITERIA["research"] existed to override this, but its formulation wasn't strong enough to suppress the base rubric's anchoring effect. The harness output scored depth=6 when the evaluator's feedback explicitly cited "missing implementation notes" — a criterion that has no meaningful interpretation for inflation analysis.

The fix added explicit negative constraints alongside research-appropriate positive definitions:

"research": (
    "This is a research synthesis task — NOT a software engineering or coding task.\n"
    "DO NOT penalize for missing 'implementation notes', 'code examples', or "
    "'named tools/APIs' — those criteria do not apply here.\n\n"
    "Reinterpret depth and grounded as follows for this task type:\n"
    "- depth: Does each section provide specific supporting evidence — data points, "
    "statistics, named regions/sectors/entities, direct quotes from sources, or "
    "mechanisms that explain the 'why'? A section that names a region but gives no "
    "figures, dates, or causal explanation is depth=6. A section with specific "
    "numbers, named actors, and causal reasoning is depth=8.\n"
    "- grounded: Are empirical claims traceable to named real-world sources — "
    "specific reports, datasets, agencies, or publications a reader could look up? "
    "Vague references to 'the Beige Book' without district or date are grounded=6. "
    "Named reports with period and district are grounded=8.\n"
)

After this fix, the perplexity output's grounded score dropped from 7 to 6 (its Beige Book references had no specific report dates or districts), while the harness output's specificity rose from 7 to 8 (its [FRED:...] inline citations are exactly the kind of traceable sourcing the corrected rubric rewards).

Final Results

Best stable result (v6 — multi-query retrieval, no extraction):

Per-dimension breakdown at v6:

The harness reaches parity through an explicit trade: it loses depth (−1.0, synthesis summarizes rather than quotes district-level specifics) but gains grounded (+1.0, structured inline citations) and structure (+1.0, better-organized output from more targeted retrieval). Perplexity's depth advantage comes from retrieving and synthesizing district-specific cost categories directly from primary source PDFs. The harness groundedness advantage comes from citation discipline enforced by SYNTH_INSTRUCTION_RESEARCH.

A Corrected Premise: Perplexity Is Also a Retrieval System

After the experiment completed, Perplexity's source panel was inspected. It listed 19 sources, retrieved live at query time. This is not training-data recall — it is web retrieval, and the source quality is high:

Several things are notable about this source list. First, it includes actual PDF downloads from district Fed websites — the Boston and Philadelphia Beige Book PDFs, the full March 5 2025 report, and the primary federalreserve.gov PDF. These are the primary sources, not summaries of them. Second, the temporal range is wider than the task strictly requires: sources include an August 2025 Beige Book summary and a January 2026 Beige Book, even though the task asks about early 2025. Perplexity pulled more temporal context than requested. Third, the Chicago Fed research paper on regional inflation differences (“housing sector is the main driver of regional inflation differences”) directly supports the causal language in the response.

Why the Depth Gap Persists

Both systems were retrieving from the same underlying corpus — the Beige Book PDFs are public documents, and the harness has indexed them in ChromaDB. If the same information is available to both, the depth gap cannot be explained by information access. It has to be downstream: how each system extracts and presents what it retrieved.

There is a retrieval fidelity distinction worth noting: embedding-search over top_k=6 chunks returns what cosine similarity ranks highest against the query embedding. It may not surface the specific passage "New York: selling price increases picked up; firms flagged coffee, eggs, freight, tariff risk" if that passage wasn't among the top-6 ranked chunks. Perplexity retrieving the full Boston and Philadelphia PDFs can scan every sentence. Same corpus, different recall coverage on specific district-level details.

But the larger factor is synthesis behavior. Given whatever district-level passages the harness context contained, the synthesis model summarized them (“regions with high exposure to trade-sensitive industries”) rather than quoting them with the district names and cost categories attached. The evaluator cited exactly this: "it could mention which Federal Reserve districts are experiencing the most persistent price pressures and provide concrete data points or quotes." The information may have been in the context; the synthesis model didn't extract it at the specificity level that scores well on depth.

The extraction experiment (v7) confirmed this. When the raw Beige Book chunks were replaced with pre-digested district/quote pairs, depth rose to 7 — the synthesis model used the structured material. But grounded fell from 7 to 6 and specificity from 8 to 7 because the extracted observations didn't carry the [FRED:...] citation anchors the synthesis model had been using from the research context. Appending both raw and extracted context (v8) didn't help: the raw context dominated and the model reverted to summarization, depth back to 6.

The tradeoff exposed by v7 vs. v8 points to the Beige Book index as the actual root cause. The price sections of district reports are chunked together with real estate, agricultural, and banking sections. Embedding similarity for "inflation" and "price pressures" returns whatever scored highest in those mixed chunks — which happened to be home sales inventory (Boston) and LNG pipeline development (Atlanta). If the Beige Book index were re-chunked to isolate each report's "Prices" subsection, the raw context itself would contain the district-level cost-category language needed for depth=7, with no extraction pass required and no citation discipline lost.

What the Experiment Validated

A falsified hypothesis is still an informative result. Several things were confirmed or clarified:

What Remains

The depth gap is a Beige Book index problem, not a synthesis instruction problem. Eight iterations of instruction tuning, retrieval sub-query design, and two-pass extraction confirmed that the synthesis model can produce depth=7 when given the right material, but cannot reliably extract district-specific cost-category language from chunks that are mostly about real estate and agriculture. Re-chunking the Beige Book index to isolate the "Prices" subsection of each district report would let the retrieval system surface the relevant material without needing the extraction pass — and without the grounded/specificity tradeoff.

The groundedness and structure advantages are real and durable: inline citations with [FRED:{series_id}:{date}] and Beige Book (Month Year, District) make claims traceable in a way that a conversational AI response is not. Perplexity produces better depth; the harness produces better attribution. For use cases where sourcing matters — compliance, audit, downstream fact-checking — the harness advantage on grounded persists regardless of the composite delta.