Seismograph – open-source early warning for silent LLM API drift
Seismograph is an open-source tool designed to detect silent drift in LLM API responses, providing early warnings when model outputs change unexpectedly. It helps developers monitor consistency in AI-generated content over time.
Background
- The project addresses **LLM API drift** — a known pain point where AI model providers silently change model behavior (e.g., tone, output format, refusal patterns) without version bumps, breaking applications that rely on stable outputs.
- **"Silent drift"** occurs when new LLM versions are deployed under the same model name (e.g., "gpt-4"), causing unexpected changes in responses, security posture, or compliance.
- **Seismograph** (by developer Tania-coder) is an open-source monitoring tool that continuously tests an LLM API endpoint against a baseline "golden answer" and alerts when behavior deviates — meant to fill a gap left by closed-source monitoring services.
- The name analogizes earthquake detection: attacks/errors are infrequent but destructive, so continuous "seismic" monitoring catches subtle shifts before they cause downstream failures.