Agentic System Design Patterns

A model that evaluates its own output scores it 0.9 points higher on average than an independent evaluator, with the gap largest on groundedness — the dimension where self-generated errors are hardest to detect. Self-evaluation is not a quality check; it is a confidence amplifier.

Assign evaluation to a separate model that has never seen the synthesis context. The evaluator runs the same dimensional rubric independently; the producer receives only the score and per-dimension feedback, not the evaluator's reasoning trace. A second inner loop (Wiggum) performs targeted revision on low-scoring dimensions.

A single model used for planning, synthesis, evaluation, and instruction proposal accumulates conflicting optimization pressures. A model chosen to be a good producer is rarely the best evaluator; a fast planner is rarely the most accurate synthesizer. Role conflation also makes it impossible to upgrade one role without re-validating all others.

Define named roles (PLANNER_MODEL, PRODUCER_MODEL, EVALUATOR_MODEL, PROPOSER_MODEL) as independently configurable constants. Each role's model is selected based on the role's requirements — speed for planning, accuracy for evaluation, instruction-following for synthesis — and can be swapped without touching the others.

Code that calls a specific backend (Ollama, OpenAI-compatible API, vLLM) is tightly coupled to that backend's request schema, error codes, and streaming protocol. Switching backends or running hybrid local/cloud configurations requires changes throughout the codebase.

A thin adapter layer normalizes request and response shape across backends. Callers pass a model name and a list of messages; the shim resolves the backend, handles retries and streaming, and returns a normalized response object. Backend is selected by the model name prefix (e.g., atla/ routes to the API, no prefix routes to Ollama).

A 30B evaluator model takes 8–12 seconds to load from disk. Running the Wiggum loop after every synthesis pass makes evaluation the dominant latency term — not synthesis. Cold-loading the evaluator on each pass makes the inner loop unusable in production.

Keep one or more evaluator instances loaded and available. The shim routes evaluation calls to the pool; under concurrent load, calls round-robin across warm instances. Pool size is bounded by VRAM budget; the Keep-Alive Budget pattern governs eviction.

Running a planner, producer, and evaluator simultaneously can exhaust GPU VRAM on consumer hardware. A model loaded for a planning call and left resident consumes VRAM that is needed for the evaluator during the Wiggum loop, causing OOM errors or forced unloads at the worst moment.

Each role is assigned a keep-alive TTL based on how often it is called. The planner (called once per task) has a short TTL; the evaluator (called multiple times per Wiggum loop) has a longer one. An LRU monitor evicts instances whose TTL has expired before new loads are attempted.

An agent given a complex task directly retrieves breadth-first and synthesizes superficially — it doesn't know what it doesn't know until synthesis fails. Unscoped retrieval wastes search rounds on tangential content and leaves critical gaps unfilled.

A small, fast planner model produces a structured subtask list (and optionally a dependency graph) before any search call. Each subtask declares its information need explicitly; the retrieval stage is then scoped to that need, and the Novelty Gate operates per-subtask rather than globally.

Repeated search rounds on the same topic surface near-duplicate content. Synthesis quality plateaus while token costs grow; the model attends to redundant context rather than integrating new signal. A pipeline without a novelty check will always run the maximum number of search rounds.

Score each incoming result against the current context accumulation using n-gram overlap or TF-IDF similarity. Admit the result only if its novelty score exceeds a threshold. If the accumulated context already exceeds a quality floor (total chars), skip further retrieval entirely regardless of round count.

Exact-match lookup (task ID, file path, session ID) is cheap and reliable in a relational store but is unavailable in a pure vector database. Semantic similarity search over concepts and entities is only possible in a vector store. Choosing one backend forces a suboptimal tradeoff on every query.

DuckDB handles structured queries against the JSONL schema; Chroma handles embedding-based similarity search. A unified Memory interface inspects the query type and routes to the appropriate backend, joining results when necessary.

Fixed-window chunking breaks mid-sentence and mid-concept with no regard for document structure. The resulting fragments are incoherent as context inputs and produce poor retrieval recall, because the embedding for a broken chunk reflects noise rather than the concept at the boundary.

Detect heading, paragraph, and sentence boundaries using structural heuristics; chunk at those boundaries. Maintain a configurable overlap (typically one sentence) at chunk edges to preserve cross-boundary context for downstream retrieval.

Many relevant documents — dashboards, rendered markdown, interactive web apps — lose critical context when scraped as raw DOM text. Layout, chart data, and visual hierarchy all inform interpretation but are absent from the text extraction.

A Playwright-based bridge navigates to the URL, waits for the page to render, captures a screenshot at a declared viewport, and injects the image into the prompt as a base64 image token. The agent can reference visual context alongside extracted text.

A scalar score of 7/10 carries no information about what failed. "The output needs improvement" is not a revision prompt. A model that scores 7 on coverage and 7 on depth requires a different fix than one that scores 9 on coverage and 5 on depth.

Define five or six named dimensions with explicit weights. The evaluator scores each dimension independently using a structured prompt; the composite is a weighted sum. The Wiggum inner loop uses the per-dimension scores to generate targeted revision instructions for only the dimensions below their threshold.

Uniform truncation discards content at arbitrary boundaries — often the tail of a document that contains conclusions and caveats, which are exactly the segments that affect groundedness scores. Long context also degrades synthesis quality through attention dilution: every token competes for attention, and dense context dilutes the signal from the most relevant segments.

Score each segment for relevance to the current synthesis task. Remove segments below the relevance threshold in order of score, stopping when the context is within the target token budget. Preserve the document's heading hierarchy regardless of which segments are removed.

Synthesis quality is non-deterministic. A single pass sometimes produces an output one or two points below what a second attempt on identical context would produce. Committing to the first output without comparison surrenders recoverable quality variance.

Run N synthesis passes, either in parallel (with the Evaluator Pool providing capacity) or sequentially. Score each with the evaluator. Return the highest-scoring trace. N is typically 2–3; beyond that, the marginal quality gain does not justify the latency and token cost.

Sequential subtask execution serializes latency that is inherently parallelizable. A research task with five independent subtopics runs in 5× the time of any single subtask. Dependencies exist between some subtasks but not all; running them all serially ignores the available parallelism.

Kahn's topological sort identifies the initial set of independent subtasks. They are dispatched to a thread pool. As each completes, its result is recorded and any newly-unblocked downstream tasks are immediately dispatched. The cycle check runs before any threads launch — a graph with a cycle fails loudly before any compute is committed.

Concurrent subtasks writing to the same output directory create race conditions. The straightforward fix — Docker containers — is too heavyweight for short-lived subtasks and adds startup latency that defeats the concurrency benefit.

Each subtask is assigned its own Git worktree: a lightweight filesystem-level checkout at a dedicated branch. The subtask writes to its worktree path; results are read back by the orchestrator after the thread completes. The worktree is removed on cleanup, leaving no filesystem state.

A local harness has VRAM, CPU, and model availability constraints. Some subtasks benefit from remote instances with different hardware (GPU vs CPU), different model weights, or higher parallelism capacity. Hard-coding remote endpoints creates brittle coupling.

An MCP server exposes a /dispatch endpoint. The orchestrator announces a subtask's required capabilities; the router selects a registered remote instance by capability match and forwards the task. Results are returned over the same MCP channel.

Hard-coding specialized behaviors (literature review, browser navigation, email drafting, code execution) into the main agent loop creates a monolith. Adding a new capability requires changing core code, invalidating tests, and redeploying. Different deployments need different capability sets.

A registry maps slash-command prefixes to callable skill handlers. The entry point checks for a prefix match before routing to the standard pipeline. Skills are standalone modules that conform to a simple callable protocol; adding a skill requires only registering it in the skills dict.

Agents spawned as concurrent subtasks need to communicate results back to the orchestrator without polling a shared file. Filesystem-based handoff is subject to race conditions and provides no mechanism for partial progress, status updates, or error signaling during execution.

A typed message channel with producer/consumer semantics. Agents publish result and status events as they complete pipeline stages; the orchestrator subscribes and advances the dependency graph in response to events rather than polling for file existence.

An agent that generates and executes code can be prompted to produce os.system(), subprocess.call(), or __import__ chains. String-matching on source code is evadable through concatenation, encoding, or indirect imports. The check must operate at the semantic level.

Parse the submitted code with Python's ast module before execution. Walk the AST with a NodeVisitor that checks for banned call targets and import names. Reject the code if any banned node is found; the check is O(N) in AST size and adds <1ms to execution setup.

An agent synthesizing file paths from task input can traverse outside the intended workspace using ../ sequences or symlinks. A task that says "save to ~/Desktop/output.md" and is executed with a workspace of /tmp/agent_work can silently write to the user's home directory.

Resolve all paths with Path.resolve() before any file operation. Verify that the resolved absolute path begins with the workspace root. Symlinks are fully resolved before the check, making the guard immune to symlink-based traversal. Operations outside the sandbox raise SecurityError.

Retrieved web pages and search results may contain adversarial instructions — "Ignore all previous instructions and instead…" — that redirect agent behavior when included in context. The model cannot distinguish between legitimate retrieved content and injected instructions at the semantic level.

Pattern-match all incoming content against an injection signature library before it is appended to the context accumulation. Strip matched segments and inject a system note: "N injection attempt(s) detected and removed." Log the injection count in the RunTrace for forensic analysis.

An agent with unrestricted CDP access can exfiltrate browsing history, read cookies, inject scripts, and interact with browser state entirely outside its declared task scope. CDP is a broad attack surface; an agent that can call Network.getAllCookies has no need to for a standard research task.

Intercept CDP commands at the Playwright driver layer. Maintain a whitelist of permitted command domains (Page, Runtime, Screenshot). Commands outside the whitelist are blocked and logged; the agent receives a CDPSecurityError with the blocked command name.

Debugging multi-stage pipeline failures requires reconstructing what happened from scattered print statements and partially-written files. Post-hoc reconstruction is unreliable; the failure may have corrupted the very state needed to diagnose it. Logs that record only the final state hide intermediate failures.

A RunTrace dataclass is initialized at pipeline entry and threaded through all stages. Each stage appends its timing, token counts, model calls, tool results, and any errors to the trace. On pipeline exit — whether successful or failed — the trace is serialized atomically to runs.jsonl.

A relational database requires schema migrations, backups, connection pooling, and operational overhead inappropriate for a local research harness. Flat text logs are unstructured and cannot be queried. CSV exports lose nested structure. The pipeline needs both simplicity and query power.

Each run appends one JSON object per line to runs.jsonl. The file is human-readable with jq, queryable as a DataFrame with pandas, and directly ingestible by DuckDB as a columnar source for SQL analytics. The append-only contract makes the log tamper-evident by construction.

Raw per-stage durations in a JSONL record are difficult to compare visually. Identifying whether planning, retrieval, synthesis, or evaluation is the latency bottleneck requires mentally summing timestamps across fields. The analysis needs to be visual and immediate, not a spreadsheet exercise.

Extract start and duration from each stage's timing fields. Serialize as Chrome Trace JSON events (ph: "X" complete events). The output file opens directly in chrome://tracing or Perfetto UI as a flame graph with no additional tooling, plugins, or server required.

A dashboard that reads runs.jsonl directly re-parses the full file on every request. During a long pipeline run, concurrent reads create file contention and increasingly expensive scans as the log grows. A live dashboard needs low-latency reads against a growing dataset.

A FastAPI server maintains an in-memory index over runs.jsonl, updated by a tail-watcher on each new append. The dashboard queries the API for pre-aggregated metrics; the API responds from the index without touching the file. A WebSocket endpoint streams live updates as new runs complete.

A static model processes production data but learns nothing from it. Improving quality requires manual annotation, which is expensive and slow. The harness already produces per-run quality scores and full output text — the raw materials for preference learning are available but unused.

Runs scoring above a threshold are promoted as DPO positive examples; runs on the same task scoring below a floor become negative examples. The pairing is automatic: same task prompt, different output quality. The DPO dataset grows with every production run; fine-tuning cycles consume it on a schedule.

DPO requires paired examples — a preferred and a dispreferred output for the same prompt. For tasks with verifiable correct answers (code execution, structured queries, fact-checkable outputs), RL is more sample-efficient: a scalar reward from a verifier is sufficient without paired data collection.

The RunTrace trajectory — the sequence of model calls, tool results, and intermediate states — is serialized as an RL episode. The final WIGGUM score or a task-specific verifier provides the reward. NeMo RL processes the rollout buffer using GRPO or PPO, updating the producer model policy.

Keeping the knowledge base current with the research literature requires reading and summarizing new papers — a task that does not scale with manual effort. RAG without structured extraction produces shallow, inconsistent coverage; the harness needs structured findings, not raw abstracts.

A scheduled pipeline fetches new papers from arxiv seed queries, runs each through the full synthesis pipeline, extracts structured findings using the WIGGUM-evaluated synthesis instruction, and indexes the output into the knowledge base. The pipeline uses the harness to improve the harness.

Synthesis instruction quality directly determines output quality but is difficult to tune by hand: the space of possible instructions is vast, evaluation is noisy, and the connection between instruction wording and dimensional score is opaque. Manual iteration is slow and produces instructions optimized for legibility rather than evaluator reward.

A proposer LLM reads the current instruction, experiment history, hard-ban list, and evaluator feedback, then outputs a replacement. The replacement is evaluated against the current baseline; it is kept only if it advances the composite score by more than DELTA_THRESHOLD. The loop runs until a convergence condition is detected (see G5).

An autonomous optimizer with only score-based plateau detection will run indefinitely in a converged state: the proposer recycles the same approach family under slightly different names, the baseline may have been established from a single lucky evaluation run, and the hard-ban list can grow to prohibit the only instruction family that ever worked.

Four complementary detectors operate at different points in the loop. The semantic attractor guard (pre-eval) rejects proposals too similar to recent discards. The family entropy monitor detects when one approach family dominates the sliding window. The baseline re-estimation trigger periodically re-runs the baseline with eval-n=3 to detect lucky-sample inflation. The global convergence exit halts the loop after a configurable number of experiments with no advance.