The Most Expensive Bug Is The One That Returns Wrong Data Silently

Every team that ships a data product learns the same lesson the same way. The bug that pages you is cheap. The bug that returns 200 OK with the wrong number is the one that ends careers.

At ANIA we ingest thousands of articles a day, score them for relevance, and surface the top 1% as “signal” to VCs and founders who pay us specifically because they trust the filter. If our ingestion is down, that is loud and recoverable. If our ingestion is up but quietly misclassifying a source — say, scoring a vendor press release as original reporting — we ship a degraded product with a green dashboard. The customer does not file a ticket. They just stop opening the newsletter. Three months later we call it churn.

Loud failures vs. silent failures

Loud failures have a budget. You size your queues, you set SLOs, you wire pagers, you write postmortems. The entire SRE discipline exists to make loud failures cheap and short. Silent failures have no budget because they have no signal until the damage is structural.

The asymmetry is brutal. A 30-minute outage is a status page incident. A 30-day slow degradation is a board slide.

The five failure modes that cause silent wrong data

After a year of operating an autonomous ingestion pipeline, here are the five failure modes we treat as existential. None of them throw.

1. 1. Fan-out joins that multiply the wrong dimension

   You join articles to entities on a key that is no longer unique after a schema migration. Row counts look healthy. Aggregates double-count. The top-articles chart now ranks cross-posts as if they were independent scoops. Nobody notices because the chart looks plausible.

2. 2. Scorer drift on a quiet feature

   A relevance model depends on a feature sourced from a third party. The third party silently changes their scale from 0–1 to 0–100. Your model did not retrain. Relevance scores now skew 60 points high. The dashboard says signal quality is up. It is not.

3. 3. Time-zone naive timestamps

   A source switches their feed from UTC to Eastern during daylight saving. Your “published in the last 24 hours” window now excludes the morning rush. The newsletter misses the biggest story of the day. The next day it looks fine.

4. 4. Dedupe keys that over-match

   You tighten a dedupe threshold to cut noise. It quietly collapses three genuinely distinct stories into one. The collapsed story is now the “canonical” one. You under-report a category for a month.

5. 5. Successful retries that double-write

   A transient 5xx is retried. The first attempt actually succeeded but returned late. Now you have two copies of the same article with different IDs. Your uniqueness invariant is violated. Aggregates drift.

Why this compounds: pipeline debt

Infra debt is linear. You can see it on a chart, you can size it in dollars, you can pay it down in a sprint. Pipeline debt is geometric because each silent failure poisons the inputs to every downstream consumer. A bad score in stage three becomes a bad ranking in stage five, becomes a bad newsletter in stage seven, becomes a wrong conviction in a customer’s head. By the time the symptom surfaces, the root cause is weeks upstream and the blast radius is the whole product.

Worse, pipeline debt is invisible to the dashboards you build to manage latency. Throughput is up. Error rate is zero. Pager is silent. The system is “healthy” by every operational metric you defined. That is the trap: you instrumented for availability, not for correctness.

What we actually instrument

The shift that mattered was treating data correctness as a first-class SLO, on equal footing with uptime. A short list of what that looks like in practice:

• Golden-set regression. A frozen, hand-labeled set of ~500 articles runs through the full pipeline every hour. If precision or recall on the golden set moves more than 2 points, we page — even if the pipeline is green.
• Source-level distribution monitors. Every source has an expected distribution of relevance scores, volume, and publishing cadence. A 3-sigma shift in any of them is an incident worth investigating before it becomes a customer-facing defect.
• Uniqueness and cardinality asserts. Every join is wrapped in an assert on expected cardinality. Ifcount(distinct id)after a join is not equal to before the join, the pipeline halts loudly.
• Reconciliation against external ground truth. Once a day we reconcile our entity graph against a public, authoritative list. Drift here catches schema changes upstream providers never told us about.
• Idempotency keys on every write. Retries can never create a second row. The cost of a deterministic key is trivial; the cost of a phantom duplicate is a quarter of trust.

The cultural shift

The deeper change is in how you define a “bug.” Most teams define a bug as “the system did not do what the code said.” For a data product, that definition is too narrow. The expensive bugs are the ones where the system did exactly what the code said, the code was wrong about the world, and nobody built a check that would notice.

The reframe that worked for us: a bug is any case where the system produces an output that would change a customer’s decision if they knew how it was made. By that standard, a silently misclassified source is not a minor data-quality issue. It is a violation of the product’s core promise.

That standard is uncomfortable. It means the pipeline is never “done.” It means a green dashboard is a hypothesis, not a verdict. It means the most expensive line on the engineering budget is the one nobody is forced to write: the assertion that catches the silent wrong answer before it leaves the building.

A 200ms delay costs you a pager. A silent wrong-row join costs you a quarter. Build the asserts that make the second one as loud as the first.