1. Overview
Large enterprise portals containing thousands of API documentation pages frequently suffer from outdated code snippets, broken external links, and formatting discrepancies. Manual quality audits across these portals were labor-intensive and error-prone.
The Doc Portal Reviewer combines headless browser automation (Playwright) with LLM evaluation to crawl pages autonomously, evaluate content against defined technical checklists, capture screenshot evidence, and produce structured audit reports.
2. The Problem
- Scale: Hundreds of pages cannot be audited manually before each release cycle.
- Subtle Deficiencies: Syntactic errors in code snippets or misaligned API schema versions often slip past basic link-checkers.
- Evidence Documentation: Manual reporting required engineers to capture and upload individual screenshots manually.
3. System Architecture
4. Technical Implementation
- Headless Playwright Automation: Handles single-page application (SPA) client rendering, authentications, and dynamic navigation states.
- Confidence Scoring Engine: The LLM returns a confidence score (0.0 to 1.0) along with its evaluation verdict. Items falling below confidence thresholds are flagged for human review.
- Pluggable Providers: Supports multiple model backends (AI Canvas, OpenAI, or local Ollama instances) depending on deployment constraints.
5. Results & Impact
- Automated end-to-end documentation audit sweeps across enterprise documentation sets.
- Produces HTML and JSON reports that an engineer can review before documentation is released.
Evaluation & limits
LLM confidence is a model output, not a calibrated accuracy measure. Low-confidence findings are intended for human review. The public page does not publish a labeled audit set or a false-positive rate.
A next step is to compare generated findings with reviewer decisions across a sanitized document set.