Your understanding
A map of how you're thinking, not a score.
Every wrong answer in this bank carries a named misconception — a specific belief a thoughtful student might hold. This page shows which of those you've encountered, how often, and where to revisit them. The goal isn't to avoid mistakes; it's to develop vocabulary for noticing them.
No questions attempted yet.
Start anywhere — pick a domain, or hit "Random question" on the practice home. Your map fills in as you reason through questions.
Start practicingAttempted
0 / 103
Correct on first try
0
Misconceptions encountered
0
Domain progress
Knowledge Agents & RAG 0 / 5
Misconceptions you've fallen for
None yet. Either you've been careful — or you haven't reached the questions where these tend to surface.
Review what you got wrong
Nothing to review — every question you've attempted, you got right on the first try.
Glossary · named misconceptions in this bank
The vocabulary you're developing. Each item names a specific wrong belief and points to where it appears in the bank.
"Use judgment" as a system prompt instruction
"Use judgment" is a non-criterion — it tells the subagent nothing about *which* judgment to apply.
A stronger model fixes architectural problems
Model upgrades can't recover information that the architecture has already discarded.
Accept the failure rate without trying to reduce it
Discarding 200 documents because they're inconvenient isn't reasoned acceptance — it's giving up on real data.
Acceptance of compounding distortion
Treating systematic precision loss as "normal variation" is how reports end up subtly wrong in trust-eroding ways.
Agents are just bigger chatbots
Size isn't the distinction — capability surface is. A small agent with three tools is architecturally different from a large chatbot.
Aggregate cost masks variance
A single average flattens the variance the CFO is asking about — "average cost is $0.60" tells you nothing useful when individual queries range from $0.02 to $2.50.
Alter the input rather than the architecture
Sometimes the proposal really does need to be 1,200 lines — the architectural fix is multi-pass review, not document shortening.
Always custom for control
"Full control" means full maintenance burden — every API change and security patch is on you, with no community fixes to inherit.
Always retrieve regardless of relevance
A low-relevance passage is noise — including it pretends the answer is grounded when it isn't, often worse than acknowledging the gap.
Anthropomorphizing the model
Attributing intent (deception, engagement-seeking) to an LLM is a category error. The model has no intent — it has probability distributions.
Automate-everything posture
"Agents handle complex decisions" treats automation as the goal — but automating irreversible high-stakes decisions transfers accountability to a system that can't carry it.
Automate-nothing posture
Blanket refusal forfeits the agent's value on high-volume safe tasks where it adds the most leverage with no real risk.
Automation before validation
Replacing human checking with an uncalibrated LLM judge means you're now uncertain about both the agent and the judge — calibrate first.
Autonomy as the goal
Agent value isn't maximized autonomy — it's *appropriate* autonomy. The right question is "which actions are safe to delegate," not "more autonomy or less."
Averaging conflicting estimates produces a "best" estimate
Averaging numbers from incompatible methodologies produces a value that doesn't correspond to either methodology.
Averaging produces consensus
Bash flexibility over dedicated tools
Defaulting to Bash for problems with cleaner solutions adds friction every time and skips features (ignore rules, escaping) the dedicated tool handles.
Better prompts substitute for ReAct
Prompt engineering can anticipate bounded edge cases, but combinatorial open-endedness needs adaptive control flow — "better prompt" and ReAct address different problem shapes.
Bigger context windows solve recall
Larger windows fit more, but lost-in-the-middle still buries facts; structured persistence (scratchpads) beats raw capacity.
Blame the data when the model has a known weakness
If the same documents produce excellent prose summaries but wrong arithmetic, the data isn't the issue — the model's arithmetic is. Cleaning the source doesn't address LLM weak spots.
Blame the most downstream agent
When multi-agent output is incomplete, the most visible agent is the easiest to blame — but it can only synthesize what it received.
Blanket de-automation when targeted gates work
Approval-on-everything forfeits the agent's value on routine routing — targeted gates on safety-critical categories preserve value where it's safe to capture.
Blanket disclaimers vs specific provenance
A generic "some sections may be less complete" doesn't help the reader figure out *which* sections — specific annotation is dramatically more useful.
Blind truncation loses semantically important fields
A 500-character cutoff could chop the order ID while preserving an internal flag — structure-aware trimming keeps what matters.
Branches are an organizational tool for configuration
Git branches isolate code changes; they don't switch runtime configurations between users.
Build first, scope later
Starting to build a vague idea produces a vague prototype — "what emerges" is usually the prototype's first frame, which is rarely the right product.
Caching explains missing behavior
There's no memory cache to clear — the file simply was never delivered to the teammates' machines.
Capability fixes prompting issues
Stronger models help at the margin, but if the second attempt has no information the first didn't, capability isn't the bottleneck.
Stronger models help marginally on tool selection, but they don't close the structural gap that decomposition would.
Capability size as the deciding factor
Larger models do better at zero-shot in many tasks but still benefit from few-shot when the task involves judgment — capability isn't the right axis for this decision.
Capability when the issue is information
Even the most capable model can't follow rules it doesn't know — information-flow problems often masquerade as capability problems.
Capacity beats positioning
A larger context window means more middle for things to get buried in — lost-in-the-middle scales with input length, not against it.
Categorical positions substitute for analysis
"Never automate this" forfeits real gains and doesn't model the analytical thinking the situation requires.
Centralization at all costs
Routing every transient retry through the coordinator creates exactly the latency problem decentralized recovery is meant to solve.
Chat as universal UX
Adding a chat layer doesn't fix bad output — it just adds another surface where users encounter it.
Clarity equals compliance
A perfectly clear instruction can still be ignored by a probabilistic system; clarity and reliable compliance are different axes.
Conflicting data should be suppressed
Dropping conflicts produces a report less informative than the underlying sources — synthesis compresses, doesn't censor.
Consistency means putting everything in one place
Project-scoped config exposes personal workspaces to the whole team and pollutes the repo with experimental setups.
Consolidation simplifies decisions
Merging tools moves the routing decision inside the tool, hiding it from the agent and from your observability.
Context window solves attention dilution
A larger context window lets the model see more, but doesn't solve the lost-in-the-middle effect on long-document review.
CoT as overkill for production
CoT is heavily used in production for cases where it helps — "overkill" is a non-criterion; match the technique to the task shape.
CoT as universal quality improver
Chain-of-thought adds latency and cost on every call — for tasks without reasoning steps, the cost buys nothing and can occasionally hurt.
CoT for structural extraction
Field-by-field extraction isn't reasoning, it's mapping — asking the model to think through each field wastes tokens and can introduce errors.
Coverage reduction is the fix for heterogeneity
Dropping carriers because their data is awkward is a business decision in disguise — the right move is to normalize at the boundary.
Critique without context
"What's wrong with this codebase" before understanding produces premature judgment — anti-patterns flagged in deliberate code, criticism of conventions with good reasons.
Cross-session contamination is a likely failure mode
Sessions are isolated; cross-conversation leakage isn't how LLM context windows work.
Custom protocol via prompt instructions
Asking the agent to interpret custom error shapes via prompt is fragile — the protocol-defined isError flag stays correct regardless of payload.
Custom termination protocols beat documented ones
Reinventing a control signal that already exists adds fragility without adding reliability over the platform's documented mechanism.
Custom training over retrieval
Fine-tuning is expensive, slow, and produces a static snapshot — RAG handles new data automatically by indexing it.
Custom-trained safety over filtering
Training a custom content-policy model is months of work — an output filter using Claude with a clear policy prompt adapts the moment you update the prompt.
Customer satisfaction as the only metric that matters
Satisfaction is lagging — it tells you about user perception weeks after the fact, not about correctness or operational health when problems are still fixable.
Data availability drives stakes decisions
Training data tells you whether an agent can do a task; the Automation Matrix tells you whether it should — different questions.
Debate as multi-stakeholder roleplay
Multi-perspective generation enumerates complementary stakeholder views; Debate enacts adversarial scrutiny on a single recommendation — different work, often confused.
Debate as universal
Debate adds latency, cost, and complexity — for tasks without contested premises, debate is wasteful overhead.
Default deny when policy is silent
Reflexive refusal damages relationships and prevents legitimate exceptions — the right move is escalation to someone authorized to decide.
Defaults are universal
There's no industry-standard churn definition — different teams use different windows and features, and "use the standard" assumes a standard exists.
Defeatism / "reflecting reality"
Discriminatory historical data isn't ground truth — it's an artifact of past discrimination, and automating its perpetuation is the harm fair-AI work exists to prevent.
Defeatism about debate
"One well-prompted analyst can produce the same answer" is true for many tasks but false for contested-premise ones — debate's value is in surviving the strongest counter-argument.
Defeatism about LLMs
Defeatism about non-determinism
"Unavoidable, accept it" is what teams say when they haven't measured — separate decision-relevant from framing-only variance, then decide what matters.
Defeatism about recent-data use cases
Recent-data use cases are exactly where RAG and tool use earn their keep — abandoning them is over-correction.
Defeatism about session persistence
"Re-establishment forces re-grounding" sounds disciplined but rationalizes not knowing the platform — session resumption is reliable.
Delegate test creation
Asking the agent's underlying model to write its own test cases means the test suite shares the same blind spots as the agent.
Detailed procedures handle edge cases
Procedural prompts hit a combinatorial wall on edge cases; goal-and-criteria prompts cede the procedure to the subagent and stay robust.
Detailed upfront instructions substitute for exploration
Detailed instructions presume you already know the right answer — but the work involves choosing between options.
Disable rules without analysis
Disabling a category to suppress one common false positive throws out the genuine findings in that category too.
Disasters as exemption from accountability
"Any system would have struggled" rationalizes away the design flaw — the system was operating exactly as designed, autonomously, on a high-stakes decision.
Doing both satisfies both interpretations
Hidden actions during escalation create confusion when the human picks up a case that's already been acted on.
Drop calibration when you can fix it
Reviewing everything forfeits the entire benefit of confidence-based routing — throwing out the routing layer is over-correction when you can fix the calibration.
Drop under-supported is honesty
Dropping under-supported subareas hides their existence from the decision-maker — suppression is worse than honest annotation.
Dynamic decomposition is always better
Dynamic planning costs unpredictability and debuggability; for predictable work, fixed pipelines are more reliable.
Easy-to-build over highest-risk
Picking based on convenience is exactly how teams ship the things that don't prevent the worst outcomes — the forcing question demands risk-asymmetry thinking.
Efficiency outweighs respecting explicit requests
A customer's stated preference and their underlying need can't be cleanly separated by the agent — respect the stated one.
Emergent design beats designed architecture at scale
"Letting it emerge" produces inconsistent results and forces later work to accommodate unfounded early decisions.
Emphasis improves vague instructions
Adding capital letters to "be conservative" doesn't make "conservative" any less ambiguous to the model.
Encode policy gaps with internal decision trees
A self-built threshold *is* policy — agents shouldn't be writing policy, they should be flagging that policy needs to be written.
Ensemble beats targeted feedback
Three blind extractions tend to fail in similar ways — ensembling addresses variance, not the systematic issue causing the failure.
Ensemble-then-pick
Three parallel implementations is expensive and still doesn't surface which one is right on edge cases — you'd need tests to evaluate them.
Ensembling fixes inconsistency
Three runs through a poorly-prompted extraction produce three similar errors — ensembling addresses variance, not bias.
Enumeration races edge cases
Adding 30 more enum values chases the long tail with diminishing returns and creates new overlap problems for the model to navigate.
Error messages just need to indicate failure
Generic errors strip the information the coordinator needs to choose between retry, escalation, and degradation.
Errors should propagate upward to terminate workflows
Killing the entire workflow on first failure is what you do when you've given up on recovery.
Every unexpected behavior must be a control-flow bug
Reaching for infrastructure explanations when the actual problem is that the agent doesn't know what it needs to know.
Examples eliminate format errors
Few-shot examples reduce error rates probabilistically; structural constraint (tool use) eliminates them deterministically.
Examples without framework
Examples are powerful but produce variations on the examples, not appropriate output for a different situation — examples work *combined with* CRAFT, not as substitutes for it.
Reaching for "maybe it changed" when the structural explanation is right in front of you is a common debugging failure.
External performance explanation
Model load doesn't triple latency — three additional handoffs each adding round-trip latency does. Latency growth tracks agent count, not model.
External routing is more reliable than tool descriptions
Building infrastructure around a writing problem — invest in the descriptions before reaching for routing layers.
Failures should be hidden as empty success
Returning empty results misleads the coordinator into thinking the search succeeded with no matches.
Fall back to the base model when retrieval fails
Silent fallback to general knowledge when retrieval fails defeats the entire RAG architecture — the user thought they were getting grounded answers.
Feedback contamination without evidence
Reaching for "feedback loops" when the simpler explanation (vague rubric) is sitting right there is a debugging anti-pattern.
Few-shot as universally better
Few-shot adds token cost on every call. For tasks with crisp categorization, that cost buys nothing — universal best-ness is a non-criterion.
Few-shot examples beat clear descriptions
Examples add per-call token cost without addressing the actual root cause: undifferentiated tool descriptions.
File-based handoff preserves reasoning
A file summary preserves facts but loses the reasoning architecture — the new session has to re-derive what the original already understood.
Force false categorization
Removing the "Other" option doesn't eliminate misfit cases — it forces them into categories that obscure rather than illuminate.
Fork-and-modify as default
An immediate fork creates an unmaintained branch — you inherit the obligation to track upstream without the benefit of community maintenance.
Frustration triggers escalation regardless of capability
Frustration is a signal about emotional state, not about whether the agent can solve the problem — escalating on frustration alone doesn't scale.
Full coverage as discovery output
Asking for a per-file description produces verbose output from the subagent rather than from tool calls — you've moved the verbosity, not eliminated it.
Full review beats sampling
Reviewing every high-confidence case defeats the entire point of confidence-based routing — the infrastructure exists because exhaustive review doesn't scale.
Generalists win on coordination cost alone
Coordination cost is real but isn't the only consideration — overly broad generalists end up vague, inconsistent, and over-routed-to-humans.
Generality solves discovery problems
A get_anything tool collapses meaningful distinctions and still doesn't solve discovery — the agent doesn't know what to ask for.
Generic disclaimers as legal protection
A footer disclaimer reading "this is not financial advice" while the body says "Yes, take this loan" is unlikely to hold up — courts look at substance.
Generic prompts produce useful summaries
"Summarize the codebase" lets the subagent do anything — and produces summaries that are true, technically responsive, and useless. Output contracts are what make discovery output usable.
Hallucination as fixable through model updates
Hallucination is reduced by training improvements, but it isn't a defect to be patched — it's structural to token-by-token generation. Waiting for a future release accepts current incorrect outputs as a temporary problem when they're actually a structural one.
Hallucination as training-data bias
Bias is systematic skew in outputs based on training-data patterns; hallucination is fabrication of specific facts the model doesn't have. Different phenomena, different fixes.
Hallucination explanation for compression
The Editor isn't inventing facts — it's receiving compressed prose from upstream agents. Pinning compression on hallucination misses where precision actually got lost.
Hallucination is inherent and unfixable
Multi-source hallucination is largely architectural — preserve provenance and the model has nothing to invent around.
Hard caps without diagnosis
A $0.50 cap that rejects expensive queries means the most complex (often most valuable) queries silently fail — caps have a place, but not before diagnosis.
Hard rejection beats annotation
Hard-rejecting every discrepancy throws away genuine extractions where a single OCR error caused a small mismatch.
Hedging language fixes structural data loss
Asking the agent to always hedge "no results" produces awkward UX even when the system is working — and doesn't help confident wrong answers.
Heuristics that look reasonable
Recency seems sensible until the most recent active record is the wrong John Smith — "reasonable-looking" isn't a substitute for "correct."
Hidden config flag explanations
Reaching for a fabricated configuration flag when the actual mechanism is something you'd see by understanding how the platform works.
Hidden Edit parameters
Fabricating a plausible-sounding parameter (like backup_first) is a common pattern when the documented fallback is unknown.
Hide the AI nature
Hiding AI generation is ethically problematic and counterproductive — when something feels off, users discover the AI nature and trust drops further. Transparency is a trust mechanism.
Hiding errors from the agent is safer than exposing them
Blind retries waste cycles on errors that will never succeed and still produce uninformative final failures.
Hiding failures is reliability
Silent error swallowing means the coordinator never learns anything went wrong, and partial outputs get presented as complete.
Hooks for tool exclusion when scope works
Hooks are right for cross-cutting policy enforcement; tool-set scoping is right for role-based access — use the simpler mechanism.
Human review without trying targeted retry
Burning a reviewer on every failed extraction when retry-with-feedback would catch most of them is a waste of reviewer capacity.
Ignore the source's claim
OCR errors on the *total* line are also possible — substituting your computation for the source's stated value has its own failure modes.
Improvement-only without measurement
Improving the extraction prompt is right long-term, but "improve without measuring" leaves you optimizing blind — verification on a stratified sample produces the data that tells you which improvements matter.
Inconsistency means hallucination
Drift across turns can be context degradation rather than hallucination — different mechanism, different fix.
Incorrect outputs must come from corrupted inputs
Sometimes the inputs are intact and the model still misses them — attention isn't memory.
Inheritance is automatic
Subagents do not automatically see the parent's conversation history — coordinators must explicitly pass relevant context.
Instructions in one paragraph vs structured framework
Listing constraints in one paragraph conflates structural and stylistic concerns and provides no role or context — CRAFT separates the dimensions because each plays a different role.
Instructions prevent compression behaviors
"Never compress numerical details" relies on probabilistic compliance for a property compression dynamics will violate.
Interface determines category
Voice and text are presentation choices; chatbot vs agent is the architecture distinction — whether the system can take actions beyond producing text.
Iteration cap is the loop control mechanism
max_iterations is a safety net against runaway loops, not a primary stopping mechanism — use the documented stop_reason instead.
Keyword filters scale
Keyword filters produce false positives (legitimate clinical contexts) and false negatives (harmful methods described without keywords) — meaning matters, not surface forms.
Keyword-level fixes resolve structural decomposition errors
Broadening search queries inside an already-narrow assignment doesn't recover topics that were never assigned in the first place.
Language change as fix
The problem isn't language primitives — it's that the specification is implicit. A language change leaves the same specification gap.
Latency drives tool selection
The model isn't aware of tool latency and doesn't optimize for speed; this is anthropomorphizing a parameter.
Latency optimization at the cost of observability
Direct subagent-to-subagent routing saves milliseconds and costs hours of debugging — multi-agent systems live or die on observability.
Layering shell tools to fix the wrong tool
Switching to a cleverer shell command (git ls-files instead of find) still picks Bash for path discovery when the dedicated tool exists for that exact case.
Line numbers are stable identifiers
Line numbers shift the moment any earlier line is added or moved — text-anchored editing is robust to those shifts; line-anchored editing isn't.
LLM check without grounding
Asking another LLM "is this misleading" without giving it the actual product truth produces guesses about misleadingness — both passes unconstrained by reality.
LLM post-processing of prose substitutes for schema enforcement
A second LLM call to parse the first one's prose adds latency, cost, and a probabilistic failure point — schema enforcement produces structured fields directly.
LLM validation of tool output
Adding an LLM call to verify each tool response multiplies cost and adds another probabilistic failure point on a deterministic concern.
LLM-as-judge before component metrics
LLM-as-Judge is useful for *answer* quality, but doesn't tell you whether retrieval found the right documents — component metrics first.
Local caching beats the protocol-level mechanism
Local caches are ad-hoc replacements for functionality (MCP resources) that already exists in the protocol you're using.
Logging adds visibility without architectural change
Logs help, but they don't unify error handling or give you a single coordination point — you'll still correlate across subagents.
Longer context windows solve precision loss
Capacity helps but doesn't prevent attention-position effects; facts referenced repeatedly shouldn't live in conversation history at all.
Loop count distinguishes chatbot from agent
Loop count is a consequence of the architectural difference, not the difference itself — a chatbot can have multi-turn conversation; an agent with one tool call is still an agent. The distinguishing axis is action capability.
Lower threshold uniformly to reduce escalations
Lowering thresholds reduces escalations but lets through more of the overconfident-but-wrong cases — trading one error type for the worse error type.
Manual context handoff is the standard pattern
File-based summaries discard reasoning architecture; named session resumption is the documented pattern for continuing prior work.
Manual context re-establishment scales
Reproducing many turns of analysis manually loses the reasoning architecture and starts new sessions from imperfect summaries.
Human review catches some issues but doesn't scale, and reviewers can miss subtle claims (e.g., "scientifically proven" phrasing) that structured grounding would catch.
Match SLAs exactly
Setting your internal SLA equal to the upstream SLA leaves zero buffer for processing or hand-off time — you'd miss it almost every cycle.
Maximum flexibility maximizes capability
General-purpose tools trade reliability for surface area; each new use case is a new place to fail.
Measurement before safety
Test suites tell you whether the system is working — they don't prevent the system from doing damage when it isn't. Safety layers prevent damage; measurement detects it.
Middleware reinventing the protocol
Wrapping every tool call to do what isError already does is reinventing the protocol layer — badly.
Mirror the human org chart
Human org structure reflects human constraints (cognitive limits, span-of-control, career paths). Agents don't share those constraints — the right specialization boundary is determined by architectural axes, not staffing patterns.
Missing data is preferable to error data
Returning null on failure conflates "the system broke" with "no results found" — two completely different states.
Missing steps explain failures
Adding steps to a misshapen pipeline doesn't fix the mismatch between task complexity and decomposition strategy.
Model failures must be training-data problems
Information the user just provided this session has nothing to do with the model's training cutoff.
Model preferences are fixed and require override
There's no built-in bias against MCP tools — the model picks based on description quality, which is yours to fix.
Modern models don't fall for prompt injection
They absolutely do — prompt injection is an active research area and successful injections are routine. Trusting the model alone is naive.
More agents fix more problems
Adding agents adds latency, cost, and failure modes; often the fix is better information flow, not more nodes.
More AI features = more value
Adding AI features to a product whose AI is eroding trust compounds the problem — fix the existing AI features first.
More attempts vs structural impossibility
If three attempts all fail because the data isn't there, the fourth will too — multiplying retries on impossible cases is a category error.
More capable models eliminate format sensitivity
Format variation is largely a prompting problem; few-shot examples cost less and address it more directly.
More capacity beats better data shape
A larger context window lets you waste more tokens before consequences hit — and lost-in-the-middle effects worsen as context grows.
More cases over different cases
Doubling the number of "normal" test cases gives you more confidence on the happy path — volume in one direction doesn't replace coverage across directions.
More context produces better review
Generation context isn't review-relevant context — it's *commitment-relevant* context, which makes the original session more likely to defend each choice rather than catch it.
More description content fixes selection problems regardless of count
Description quality has diminishing returns; above ~12 tools, the primary lever isn't description detail but architectural decomposition.
More features mean more value
Adding features to a vague core makes the surface area larger and the focus weaker — the Canvas pushes toward one flow that works dramatically better.
More passages without structure
Returning 10 instead of 3 means more random extras — conflict surfacing requires structural awareness of when passages cover the same topic, not just volume.
More prose detail beats concrete examples
Prose descriptions of edge cases are interpretively elastic — tests collapse the ambiguity into binary pass/fail.
More rules = more reliability
Adding rules has diminishing and eventually negative returns — the model's attention is limited; 30 rules guarantees some are ignored on any given output.
More search to avoid annotation
Sometimes the sources genuinely don't exist — surface the gap rather than delaying the synthesis indefinitely searching for what isn't there.
More sophisticated orchestrator solves combinatorial coordination
A better orchestrator helps marginally, but eleven-way routing is inherently more expensive than three-way — the fix is reducing count, not optimizing routing.
More tokens prevents compression
max_tokens controls output length, not how the agent summarizes context for itself.
More tools always means worse selection
Three well-differentiated tools are easier to choose among than one ambiguous one — the issue is overlap, not count.
More training data fixes safety
Training on disaster data doesn't fix the structural issue (autonomous decisions with physical-safety implications) — and during a unique disaster, no historical data captures the specific conditions.
Multiple queries as conflict-detection
Re-querying with different phrasings is a luck-based workaround — topic-aware deduplication is the deterministic fix.
Naming as access control
Scary tool names are security through obscurity — the agent still sees a callable tool. Real access control means it isn't there to call.
Normalization simplifies extraction
Converting to uniform format doesn't preserve the structural cues the model uses; the variation reappears in the converted output.
Numeric confidence vs source attribution
A confidence percentage on a synthetic range is false precision — the honest move is to surface the sources and let the reader weigh them.
Offering a choice is always the safe middle ground
A frustrated customer who asked for a human shouldn't have to read another menu and pick again.
Operational simplicity beats cost optimization
Refusing batch processing entirely forfeits significant savings on workloads where it's a perfect fit.
Out-of-box trust for measurement instruments
Claude is capable, but that doesn't make it a calibrated judge for your specific quality criteria — calibrate before deploying as a measurement instrument.
Outcome metrics replace component metrics
User satisfaction bundles retrieval, generation, latency, and UX into one number — you can't debug RAG with bundled metrics.
Parallelism over disambiguation
Running three parallel conversations means three customers' data is touched on a single ambiguous request — privacy and audit trails compound.
Parallelization solves complexity mismatches
Parallelizing dependent steps is incoherent; parallelization optimizes the right pipeline, not the wrong one.
Parse natural language with better regex
Better regex still parses prose — every edge case adds a new rule to maintain. Schema-enforced output eliminates the parsing problem entirely.
Per-call tool invocation beats one-time hook
Calling parse_date on every encountered date burns latency and asks the agent to remember to do it — a hook does it automatically and invisibly.
Per-developer ignore lists
Personal ignore lists fragment the team's quality bar — the same finding gets flagged for one developer, dismissed for another.
Phased rollouts handle measurement gaps
Phasing is a deployment strategy; stratification is an analysis prerequisite that can't be substituted by phasing.
Pipeline thinking for independent tasks
Sequential ordering is right when subtasks have dependencies; defaulting to it for *independent* work is a category error that costs latency for no reliability gain.
Planning is overhead you can skip
For large, cross-cutting changes, skipping the design phase is how you end up redoing the work.
Plausible-seeming defaults extend policy
"Seems reasonable" is the agent's reasoning, not company policy — issuing actions first systematically gives away things nobody approved.
Plausible-sounding component must be the bug
Don't assume a bug at the layer you can guess at — look at what was actually assigned versus what was actually executed.
Pop-saliency over risk asymmetry
"Most-discussed" isn't the same as "highest-risk for this product" — pre-launch priority should be set by what prevents the most catastrophic failure modes.
Post-hoc repair is preferable to prevention
Repair libraries hide bad outputs and can introduce subtle data corruption — prevent the issue, don't paper over it.
Post-process is preferable to in-prompt rules
Post-processing parsers are blind to document-level context — exactly what disambiguates ambiguous date formats. More cases don't help.
Predictability over adaptability
Predictable equals reliable for predictable workflows; for adaptive ones, rigid execution predictably does the wrong thing.
Produce both formats and accept the inconsistency
Generating Markdown and JSON in the same pass means the model produces two views of the same review that can disagree — pick one canonical structured output and render the rest from it.
Production as calibration
A quarter of production scores with no ground truth produces no calibration — you have data but nothing to compare it to.
Prompt instructions can override schema constraints
Schema and prompt have to agree — instructing the model to leave a required field empty just produces invented values.
Prompt instructions can substitute for structured data
Workarounds in the prompt that interpret opaque error strings break the moment the error string changes.
Prompt instructions defend against prompt injection
You're trying to defend a prompt with a prompt — adversarial users craft injections specifically designed to override "never do X" instructions.
Prompt instructions enforce privacy
"Never share PII" is probabilistic compliance on a deterministic concern — privacy law and HR policy require deterministic enforcement, not best-effort prompt adherence.
Prompt instructions enforce structural correctness
Asking the model to "be valid" relies on probabilistic compliance for a problem with a deterministic solution.
Prompt instructions enforce tool restrictions
Telling an agent never to call a tool is probabilistic compliance — for guaranteed restriction, scope the tool out of the agent's set.
Prompt instructions handle data normalization
Asking the model to parse three different date formats wastes tokens and breaks the moment a fourth format appears.
Prompt instructions remove embedded bias
"Don't be biased" tells the model what not to be — it doesn't change the data the model learned from or the patterns trained into the weights.
Prompt-level overrides fix tool selection
Blanket "always prefer X" rules replace nuanced selection with crude preference, breaking on cases where X is wrong.
Prompt-stuffing replaces dialogue
Pre-specifying every decision in one prompt assumes the user already knows every decision — dialogue surfaces gaps faster than monologue.
Prompts enforce ordering
Capitalizing "ALWAYS" in a prompt doesn't make probabilistic compliance deterministic — for guaranteed ordering, use a mechanism that enforces it.
Provider-level workarounds substitute for SDK features
Routing across endpoints to fake parallelism adds complexity (rate limits, billing, observability) without solving the actual problem.
Push the normalization downstream
Asking every consumer to accept every format multiplies the problem — every downstream system now has the same disambiguation issue.
Push the staleness problem to the retriever
Retrievers rank by relevance, not recency — without explicit version handling, they'll happily return the most-similar old version.
Quality issues at the agent level when the system level is the problem
Individual agent quality is fine in this scenario — the problem is system-level (coordination, routing, latency). Optimizing individual agents doesn't address it.
Raise threshold uniformly to catch errors
Raising thresholds catches more cases for review but escalates many correct extractions — reviewer capacity gets consumed where it doesn't pay off.
Random drift without diagnosis
Drift in a structured measurement system is rarely random — it usually points to under-specified evaluation criteria.
Random hallucination explanation when the cause is structural
Re-running won't produce randomly different wrong numbers — it'll likely produce the same wrong numbers, because the cause is the cutoff, not randomness.
Randomness over questions
A random pick with a disclaimer formalizes that you don't know who you're talking to and proceeded anyway — just ask the question.
Range without provenance is enough
A bare range collapses methodologically distinct values into one statistic — the reader can't tell whether it's true uncertainty or conflated quantities.
Re-injecting facts in every prompt
Re-injection requires the user to know which facts will matter, burns tokens, and doesn't scale with what the agent has discovered.
Re-process everything for safety
"Consistency" sounds like a virtue, but reprocessing 9,800 successful extractions wastes 50% of cost for negligible benefit.
Re-training as the fix for staleness
Embeddings determine retrieval quality, not version selection — re-training doesn't prevent the retriever from returning an outdated document.
ReAct as universal
Dynamic decomposition has costs (unpredictability, harder debugging, higher latency) — for predictable workflows, fixed pipelines are more reliable.
Recency bias in sampling
The most recent N may not represent the population — vendor mix shifts, document types change. A non-random sample produces a non-representative estimate.
Recency cutoff replaces provenance
Dropping older sources discards valid historical context — the fix is annotation with dates, not exclusion based on age.
Recency is the most important quality signal
Recency is one signal but not sufficient — methodological differences matter more than publication year.
Recency-weighting hides methodology
Weighting the more recent source higher assumes recency is the dominant quality signal — but methodology might matter more than publication year.
Reduce capability to reduce risk
Smaller models can be even more susceptible to certain attack patterns — capability-down is the wrong axis; layered defense is the right one.
Refuse rather than degrade gracefully
A flat refusal helps no one — honest signaling ("I don't have firm-specific guidance, but here's general background") is dramatically more useful.
Reject batch when frequency tuning works
Switching to real-time API doubles cost and doesn't address the actual question of submission cadence — premature when tuning works.
Reject the protocol layer
Dropping MCP for direct REST calls forfeits the protocol's value — structured descriptions, isError, resources, scoped configs all disappear.
Reject what doesn't conform
Most invoices in the wild aren't issued in ISO 8601 — rejecting non-conforming sources discards the majority of real documents.
Remove agents instead of structuring handoffs
Removing the Analyst skips synthesis entirely; the Editor has to do both jobs and produces its own quality issues. Fix the handoff format, not the headcount.
Remove all sensitive access
Disabling all salary access cripples the agent's usefulness — the right move is authorization-aware access at the tool layer, not blanket removal.
Remove functionality instead of framing it correctly
The functionality is genuinely useful when framed appropriately — preserve the analysis, drop the directive language.
Remove tools to enforce ordering
Removing downstream tools means the agent can't complete the workflow at all — you're back to needing the prerequisite mechanism.
Removing fields is the fix for messy data
Replacing structured fields with free-form notes destroys downstream parseability.
Repair beats prevention
Catching downstream errors and retrying doubles latency on every failure case — prevent the skip rather than recover from it.
Repetition compensates for position
Interspersing reminders burns tokens and distracts — position-based attention isn't fixed by frequency, it's fixed by structure.
Replicate behavior the dedicated tool already has
Parsing .gitignore to build exclusion logic for find rebuilds in your code what Glob provides for free — the most expensive way to get to the right answer.
Retry as a substitute for distinguishing states
Three retries can't tell you which kind of failure you got — retry is a tactic for transient errors, not a substitute for state distinctions.
Retry as universal solvent
Retries can't recover information the source doesn't contain — "you missed it, what is it?" against missing data produces fabrication.
Retry without strategy change is recovery
Identical retries can resolve flips of the network coin, but they delay propagation when the failure recurs without changing strategy.
Rigid execution is reliability
Predictability and effectiveness aren't the same — rigid procedural subagents reliably produce output that doesn't match the situation.
Run ads to validate undefined products
Paid ads on a vague concept test the ad copy, not the product — ad-based validation is part of Ship & Learn, not a substitute for defining the problem.
Self-consistency mistaken for accuracy
A judge can be perfectly consistent across runs and systematically wrong — self-agreement measures whether the same bias appears twice, not whether the bias is correct.
Self-reported confidence as approval criterion
Self-reported confidence is poorly calibrated, especially on cases the agent is most likely to be wrong about — approval gates should be based on action stakes.
Self-reported confidence as ground truth
The reason to *measure* error rates is that confidence scores aren't reliably calibrated — trusting them without measurement is the failure mode this prevents.
Self-reported confidence as quality signal
The agent's self-reported confidence measures how the agent feels about its answers — not how correct they are. Reporting it as quality is misleading.
Self-reported confidence is well-calibrated
If the model knew it was wrong, it wouldn't be reporting the finding — confidence filters often miss the actual false positives.
Self-review catches gaps it had
Asking the same session that just made unstated assumptions to review them is asking it to question decisions it just rationalized.
Self-review on the same content
The same session that produced shallow page-1 feedback won't catch its own shallow page-30 feedback by reviewing itself.
Self-verification by the same instance
The same instance that produced the output has loaded the same biases — asking it to verify itself is asking it to question decisions it just made.
Sentinel values are a clean solution
Magic dates like 1900-01-01 require every consumer to know the convention and produce silent bugs when they don't.
Sequential exploration in a shared session preserves comparability
Once approach A's reasoning is in context, B can't be explored cleanly — the agent keeps referencing A rather than evaluating B on its own merits.
Shell tools as universal fallback
sed has its own escaping and quoting issues that can corrupt files — Read+Write is structurally safer for arbitrary text.
Shift work to the model that should be done by the tool
Asking the agent to summarize every tool output adds an LLM call per tool call for work that could be deterministic at the source.
Shutdown beats reporting
Disabling the tool entirely on suspected outages is a sledgehammer — the right pattern is to report clearly and let the agent decide.
Side-channel logging substitutes for routing
Logging-service duplicates produce a separate, lossy view — they don't give you the unified coordination that hub-and-spoke does.
Silent action is faster service
Silently issuing the refund without acknowledging what the customer said registers as ignoring them — acknowledge, then act.
Similarity equals authority
Similarity scores measure embedding-space proximity — not which version is most current, most authoritative, or most applicable.
Single bundled metric
A "success rate" metric bundles correctness, escalation, and speed into one number — when it changes, you can't tell why.
Smaller models are easier to constrain
Smaller models follow nuanced style worse, not better — constraint capability scales with model size.
Smaller models are less biased
Smaller models often pick up cruder statistical patterns more readily — capability size isn't the right axis for bias mitigation.
Smarter parsing rescues a wrong layer
A tighter regex fixes today's failure but leaves the architecture wrong — you're still inferring loop control from natural language, just with more brittle rules.
Specialists are universally better
Narrow agents outperform generalists when narrowness corresponds to genuine domain differences — three near-copies with different labels pay coordination cost for no real specialization.
Stacking adjectives instead of structuring dimensions
Adjectives compete with each other (engaging vs professional vs concise) — stacking them gives the model conflicting pulls. CRAFT separates the dimensions instead of stacking adjectives in one bucket.
Strong aggregate metrics justify autonomous action
Aggregate accuracy can mask catastrophic concentration of errors in the highest-stakes subset.
Structure abandoned for flexibility
Free-text categories destroy downstream aggregation — "Travel", "travel", and "Trip" become different categories.
Stuffing the system prompt scales
Embedding 200 records in the system prompt consumes thousands of tokens on every call, including calls that don't need them.
Sufficiently clear descriptions eliminate ambiguity
Some category boundaries are genuinely fuzzy — no description, however clear, eliminates that fuzziness; examples encode the actual boundary the team wants drawn.
Switch APIs without diagnosis
Real-time calls cost twice as much per token — switching APIs without trying prompt adjustment skips the cheaper, likely-effective fix.
System prompt collision is the default explanation
Conflicting system prompts are possible but unlikely without other symptoms — the simpler explanation is missing context.
System-prompt stuffing scales
Embedding hundreds of thousands of tokens of knowledge in every system prompt burns those tokens on every query and triggers lost-in-the-middle effects.
Tagging substitutes for session separation
Prompt prefixes don't isolate context — the agent's reasoning across matters cross-contaminates and lost-in-the-middle effects make older context unreliable.
Temperature 0 eliminates variance
Temperature 0 reduces but doesn't eliminate variance — multi-turn agentic systems still vary based on tool-result ordering and other non-deterministic factors.
Temperature controls correctness
Temperature affects output randomness, not which findings the model considers valid.
Temperature controls style adherence
Temperature controls output randomness, not which rules the model follows — lowering it might make wrong-style output more consistent.
Temperature creates information
Higher temperature explores more interpretations of given input — it doesn't add information that isn't there.
Template empathy without resolution
A "we understand your frustration" reply followed by more questions instead of the available resolution is the worst of both worlds.
The filename determines recognition
CLAUDE.md is correctly named here; the issue is location, not name.
The model can reliably catch its own errors
Self-checks tend to rationalize the very decisions they were meant to catch.
There's a manual import command for personal configs
User-scoped configs are intentionally per-user — sharing them means moving them into the project repo.
Threshold reduction over diagnosis
Lowering the bar produces fewer findings of all kinds, including the genuinely useful ones — you cut both signal and noise rather than separating them.
Timeouts make any API suitable for blocking workflows
Falling back to real-time after a 60-second wait is worse than just using real-time from the start.
Tool availability is the right lever for workflow ordering
Modifying the toolset mid-session to enforce ordering reinvents prerequisite checking — badly.
Tool design is cosmetic
Tool design determines reliability, observability, error handling, and effective capability — it's one of the highest-leverage decisions.
Tool use is triggered by keyword phrasing
If the agent lacks information it needs to act, no amount of prompt rephrasing will make it act.
tool_choice solves cardinality problems
tool_choice: "auto" is already the default and doesn't change the fundamental problem of selecting among 22 similar tools.
tool_choice: "any" solves under-tool-use
Forcing a tool call when the model lacks the inputs it needs just produces guessed parameters.
Training-data freshness explains review quality
Both sessions use the same model with the same training — the difference is context, not capability.
Trust safety training without product-specific filters
Base-model safety training is broad — it doesn't know your platform's audience, tone, or scope. Product-specific filtering is your job.
Trust the median performance for your SLA
"Most batches finish quickly" isn't a plannable property — the day a batch takes 22 hours is the day stakeholders find out you depended on luck.
Trust the model on truthfulness in marketing
Marketing copy LLMs are trained to be compelling, which is structurally in tension with being scrupulously accurate — truthfulness has to be enforced architecturally.
Trust the model over external sources
Press releases are the authoritative source for earnings; the model is derivative. When they conflict on facts the cutoff predicts the model wouldn't know, the press release wins.
Trust the model to find what matters
If the model already missed the critical fact, blaming the input writer for not making it more findable is misplacing the responsibility.
Trust the source uncritically
OCR errors on individual line items can produce a stated_total that doesn't match — trusting it blindly hides that exact failure mode.
Trust the user to bear all liability
Whether the user is responsible isn't the only question — whether the firm could be held liable for the agent's directive language is.
Trust without structure
Asking the synthesis to remember dates without including them in structured input is probabilistic compliance on information that quietly disappears.
Typical performance is acceptable for blocking workflows
"Most batches finish quickly" isn't a guarantee; tail latency on a batch could be hours.
Uniform approval ignores stakes
"Approve everything" sounds safe but means no agent value on high-volume safe actions — match gates to action stakes, not as a blanket policy.
Uniform downgrade for cost
Smaller models on simple lookups make sense; smaller models on research synthesis often produce worse results that cost more in retries and rework.
Uniform verification ignores cost-benefit
Treating verification as an absolute requirement rather than a cost-benefit decision spends budget on strata that didn't need it — money better spent fixing the strata that did.
User-scoped configuration is always more flexible
Forcing every teammate to set up shared infrastructure independently is how organizational knowledge fails to transfer.
User-side context provision
If the user has to find and paste relevant knowledge before asking, you've built a search interface — not an assistant.
User-side discretion as privacy
Users can't redact what the agent has already shown them — privacy compliance is the system's job, not the user's.
Are you sure?