RAG Observability Triad

RAG (Retrieval-Augmented Generation) systems fail in three ways: stale data, hallucinated answers, and bloated context. Most teams only measure accuracy. This framework shows what to actually track.

The Three Metrics

1. Freshness

How up-to-date is your retrieved context?

Why it matters: Stale data leads to wrong answers. Users lose trust fast.

How to measure:

• Data Age: Median age of retrieved documents
• Staleness Rate: % of queries returning docs >7 days old
• Update Latency: Time from data change to index update

Target: P95 data age < 24 hours, staleness rate < 5%, update latency < 1 hour

2. Faithfulness

Does the generated answer match the retrieved context?

Why it matters: Models hallucinate. RAG should ground answers in facts, but only if you verify it.

How to measure:

• Citation Accuracy: % of answer claims with valid citations
• Contradiction Rate: % of answers that contradict retrieved docs
• Grounding Score: Semantic similarity between answer and context

Target: Citation accuracy > 95%, contradiction rate < 2%, grounding score > 0.85

3. Footprint

How much context are you using, and is it efficient?

Why it matters: More context ≠ better answers. Bloated context wastes tokens and slows inference.

How to measure:

• Context Utilization: % of retrieved tokens actually used in answer
• Retrieval Efficiency: Answer quality per 1K tokens retrieved
• Redundancy Rate: % of retrieved docs with duplicate information

Target: Utilization > 60%, efficiency improving over time, redundancy < 15%

Implementation

Freshness Instrumentation

# Log retrieval metadata
{
  "query_id": "...",
  "retrieved_docs": [
    {
      "doc_id": "...",
      "indexed_at": "2025-10-28T10:00:00Z",
      "data_age_hours": 12
    }
  ],
  "median_age_hours": 12,
  "staleness_flag": false
}

Faithfulness Instrumentation

# Verify citations
{
  "query_id": "...",
  "answer": "...",
  "claims": [
    {
      "claim": "...",
      "citation": "doc_123",
      "verified": true,
      "grounding_score": 0.92
    }
  ],
  "citation_accuracy": 0.95,
  "contradiction_detected": false
}

Footprint Instrumentation

# Track context usage
{
  "query_id": "...",
  "context_tokens": 4000,
  "utilized_tokens": 2800,
  "utilization_rate": 0.70,
  "redundant_docs": 1,
  "redundancy_rate": 0.10
}

Real-World Example

At Google, we built RAG for ML deployment docs:

• Freshness: Indexed docs within 30 minutes of updates
• Faithfulness: Required citations for every claim, auto-flagged contradictions
• Footprint: Reduced context from 8K to 3K tokens with better retrieval

Result: 40% faster answers, 25% higher user satisfaction, 50% lower inference cost.

Dashboard Design

Build a single dashboard with:

• Freshness: Time series of median data age and staleness rate
• Faithfulness: Citation accuracy and contradiction rate over time
• Footprint: Context utilization and efficiency trends

Alert on:

• Staleness rate > 10%
• Citation accuracy < 90%
• Utilization < 50%

Measuring Success

Track these metrics:

• Freshness: P95 data age, staleness rate, update latency
• Faithfulness: Citation accuracy, contradiction rate, grounding score
• Footprint: Context utilization, retrieval efficiency, redundancy rate

Target: All metrics in green zone, improving over time