Internal Enterprise Production Engineering

5-Minute RCA Tool: AI Incident Diagnostic Agent

An internal diagnostic tool with a Python standard-library backend that combines error playbooks and repository code search.

1. Overview

In enterprise banking ecosystems, diagnosing intermittent system exceptions often required senior engineers to manually cross-reference proprietary runbook PDFs, error codes, and source code across multiple private repositories.

The 5-Minute RCA Tool brings playbook matching and repository search into one investigation flow. It proposes a cause and relevant references for an engineer to verify.

Enterprise Confidentiality Notice: Proprietary banking error signatures and internal repository names have been sanitized in this architectural overview.

2. The Problem

During high-severity incidents, manual triage suffered from high cognitive friction:

  • Error messages like generic null-pointer or timeout exceptions failed to point to the upstream offending service.
  • Runbooks were scattered across Confluence and internal PDF repositories.
  • On-call engineers spent valuable minutes grepping repository code trees across microservices.

3. Solution & Workflow

Input: Stack Trace / Error Signature │ ▼ [Signature Sanitizer & Extractor] ── strips tenant & PII identifiers │ ▼ [Curated Playbook Matcher] ── semantic similarity over verified runbooks │ ▼ [GitHub Code Search Agent] ── live regex & symbol search across linked repos │ ▼ [LLM Diagnostic Synthesizer] ── formats root cause, verified fix, & file paths │ ▼ Output: Structured RCA Briefing (Root Cause + Fix + Code References)

4. Technical Decisions

  • Zero-Dependency Lightweight Backend: Engineered in standard Python without heavy orchestration frameworks, ensuring rapid startup and zero operational baggage.
  • Evidence-first output: The diagnostic prompt asks for file and playbook references and should return no match when evidence is insufficient. An engineer still checks the proposed cause.
  • Security Guardrails: Sensitive credentials, API keys, and customer data are scrubbed before analysis.

5. Impact & Results

  • Combines recurring manual search steps in an internal engineering workflow.
  • Returns a structured briefing with candidate cause, possible fix, and references for review.

Evaluation & limits

The public case study omits proprietary examples and has no published time-savings benchmark. Code search can miss a cause outside the indexed repositories; proposed fixes require engineer verification.

A useful next step is to compare diagnoses with resolved incidents, tracking helpful matches and false leads.