Deterministic first
Detectors and correlation do not wait on a model. If generation is unavailable, the incident still exists.
Grounded incident intelligence
The war room,
Logs and metrics enter as files. They leave as a ranked incident, a cited RCA hypothesis, and a report an on-call engineer can actually hand to a VP. No containers. No local model. Navy war-room, amber action, built like a product company.
14
Anomaly families in the checkout cascade
5m
Default timeline bucket
85%
Coverage gate on the Python core
0
Local model runtime required
Manifesto
Most incident demos skip the layers that make generated analysis trustworthy.
We built the boring machinery first: typed ingestion, UTC timelines, MAD detectors, a dependency graph, evidence ranking, and grounding checks. Generation is the last mile, not the product.
The console is designed the way Linear, Cursor, and GitHub Copilot are designed — dark density, generous type, one accent for action, another for signal. Amber for action. Ice for telemetry. Navy from a 1am war room.
Bring a CSV. Leave with a hypothesis you can argue with.
Cited or unmarked
Facts carry evidence ids. Inferences are labeled. The UI never presents a paragraph as gospel.
Degrade, don't die
Missing logs, missing metrics, and provider failure become warnings in the run summary, not a blank page.
Artifacts you can audit
Every run writes JSON the same way a compiler writes IR. Reviewers can disagree with the machine, in public.
Scope
Investigation after the page. Not a second paging product.
Not paging
PagerDuty, Opsgenie, and the phone stay where they are. This is the investigation after the page.
Not a metrics warehouse
It does not store months of time series. Bring a window. Leave with a hypothesis.
Not a public SaaS
The API is for a trusted workstation. There is no end-user auth. Do not bind :8000 to the internet.
Not a chatbot
There is no prompt box on the console. You run a pipeline. You read a report. You approve or reject it.
Live console
The board is running.Watch it work.
A looping capture of the real console. The pipeline runs against the live API when it is up, then the board fills with jobs, incidents, and a grounded report.
- 1. Health pill is live against 127.0.0.1:8000.
- 2. Checkout cascade is the noisy, production-like path.
- 3. Facts stay attached to detector rows — not model prose.
incident · console
rec 0:00api checkingCommand center
Tonight's board
Jobs
0
Incidents
0
Anomalies
0
Checkout cascade
cascade
Healthy baseline
healthy
Degraded partial
degraded
Waiting for a scenario run.
autoplay · loop0%
Eight stages
A compiler for incidents, not a chatbot with logs pasted in.
01
Ingest
CSV, JSON, and JSONL become typed log events and metric points. Invalid rows are counted, not silently dropped.
02
Normalize
Timestamps lock to UTC. Signals align into five-minute buckets so detectors share one clock.
03
Detect
Latency, error rate, CPU, memory, traffic, and availability each have their own z-score and MAD gates.
04
Correlate
Anomalies group by time, service, and the dependency graph. Isolated noise stays isolated.
05
Rank evidence
RCA scores origin vs blast radius. Downstream pain is not automatically the cause.
06
Ground
Every claim is checked against detector output and retrieved runbooks. Unsupported prose is marked.
07
Compose
The report is assembled from ranked evidence. Optional Groq or OpenAI can rewrite; the facts never leave the bundle.
08
Review
Draft, reviewed, approved, rejected. Approved reports can leave through a webhook with an audit log.
Case 001 · checkout cascade
One service. Five signals. Zero mystery theater.
The bundled sample is a saturated checkout path: CPU, memory, latency, error rate, and an error-log burst arriving in the same five-minute windows. Correlation collapses them into a single incident. RCA names checkout-service because the evidence is there — not because a model felt poetic.
Run it in the console →checkout-service CPU 96.0 against a 35.0 baseline
error rate 0.22 against 0.012
p95 latency 1900ms against 125ms
memory 1200 against 425
error log burst with no counterpart on the healthy window
traffic drop on the same five-minute bucket
service unavailability flagged beside the saturation cluster
one correlated incident, not seven disconnected alerts
Paste vs pipeline
A chat box is not an incident compiler.
Chat paste
- Paste a CSV into a chat box
- Ask a model what went wrong
- A fluent paragraph with no citations
- A chatbot that will invent a root cause
- Restart the session and lose the thread
- Hope the next shift believes you
This pipeline
- Validate every row against LogEvent and MetricPoint
- Fire eight independent detectors, then rank origin on a graph
- Claims mapped to evidence ids; unsupported sentences stay marked
- Closed-book Q&A that cites this run's pack or refuses
- A timestamped artifact directory you can re-open tomorrow
- Review states, reviewer names, and webhook delivery JSONL
Who it is for
Four chairs in the same war room.
On-call engineer
Need a ranked origin in minutes, not a wall of Grafana panels. The console names the service, lists facts, and writes the handoff you would have typed anyway.
Incident commander
Need a single candidate, not seven alerts. Correlation score, blast radius, and review state live on one board so you can decide who to page next.
Staff SRE / platform
Need artifacts you can regress. Golden evaluation summaries, coverage gates, and schema contracts keep the pipeline honest across releases.
Engineering manager
Need a paragraph a VP can read. Executive summary, facts, and uncertainties are separated so nobody has to decode detector jargon in a staff meeting.
Detectors
Eight families. Independent gates. No silent fusion.
Each detector reads its own series, applies its own z-score and MAD thresholds, and emits an AnomalyCandidate. Correlation happens later. That order is the whole product.
FamilySignalGate
error_rate_spikeHTTP 5xx / error_ratez ≥ 2.5 · MAD × 2.5 · Δ ≥ 15%
latency_spikep95 / request_latency_msz ≥ 2.5 · MAD × 2.5 · Δ ≥ 20%
cpu_anomalycpu_usage / cpu_percentz ≥ 2.5 · lookback 20 buckets
memory_anomalymemory_usage_mbz ≥ 2.5 · relative change 15%
traffic_droprps / throughputz ≥ 2.2 · MAD × 2.2
service_unavailabilityupstream_failure / unavailablez ≥ 2.0 · higher severity weight
error_log_burstERROR log countburst vs baseline · min support 3
critical_log_burstCRITICAL log countseverity weight 1.2 in correlation
A shift
Five moves between the page and the handoff.
01
Confirm the clock
API health on :8000, then a bundled scenario. Checkout cascade is the noisy path. Healthy baseline is the empty-ledger control.
02
Read one candidate
Fourteen detector hits should collapse to one incident. If they do not, correlation failed — that is a bug, not a preference.
03
Argue with the origin
Primary service is a ranked guess. Evidence rows are facts. Contain the origin, not every downstream waiter.
04
Write the handoff
Paste the engineering section into the ticket. Facts stay attached to detector ids so the next shift does not re-litigate the window.
05
Close the state machine
reviewed, then approved or rejected with a name and a note. Only approved reports may leave through a webhook.
From the floor
“I used to screenshot six Grafana rows and write the same paragraph. Now I run the cascade and edit the handoff.”
“The correlation score is the only number I read first. If it is one incident, we stay in one bridge.”
“I will not ship a detector I cannot regress. The golden summary is the contract, not the landing page.”
“The executive section is the only thing I paste upstairs. Facts stay downstairs where they belong.”
“Missing metrics used to blank the page. Now the run summary warns and the detectors that still have support keep firing.”
“Healthy baseline is the control I wanted. If that ledger is empty, I trust the noisy run a lot more.”
Evaluation
Modes you can regress, not vibes you can demo once.
ModeIncident F1Root causeNote
heuristic-only0.9476.7%Detectors and RCA without generated prose.
grounded-compose0.94pipelineSame analysis, narrative composed from the evidence bundle.
openai-optionalopt-inopt-inRemote rewrite. Never the default path.
mock-llm-retrievalharnessharnessEval-only path for citation and completeness gates.
In
Files, a graph, a corpus.
Logs
CSV, JSON, or JSONL with timestamp, service, severity, and message. Invalid rows increment parse failures instead of vanishing.
Metrics
The same file types, plus optional Prometheus query_range. Series names map through configs/default.yaml — latency, CPU, memory, error rate, traffic, availability.
Graph
configs/service_dependencies.yaml. Edges tell RCA which pain is downstream and which service is allowed to be the origin.
Knowledge
Markdown runbooks and historical incidents under data/knowledge. Retrieved as snippets with citation ids, never as unbounded memory.
Out
Artifacts you can argue with.
JSON artifacts
Timeline, anomalies, incidents, RCA, grounding, reports, run summary. The compiler IR. The UI is a viewer.
Markdown / HTML
incident_report.md and .html for the ticket and the war-room screen. Same facts, different surface.
Webhook audit
Approved reports may POST to an allowlisted URL. Attempts append to exports/webhook_deliveries.jsonl even when they fail.
Eval summaries
incident F1, root-cause correctness, claim support rate, unexpected service mentions. Compared against eval/golden.
Quality
Gates that fail the build, not the landing page.
pytest + coverage
85% coverage gate on the Python core. Detectors and RCA have adversarial fixtures.
ruff + mypy
Lint and strict types on every contract the pipeline emits.
eval regression
compare-eval against golden summaries. A detector change that moves F1 fails CI.
CodeQL + Dependabot
Static analysis and dependency review on the same cadence as the product UI.
Limits
Honest edges, written down.
Trusted workstation
No end-user auth. Path allowlists and URL policy are foot-gun guards, not a multi-tenant security model.
Jobs are memory
Restarting uvicorn clears the board. Artifacts remain. Re-run a scenario to refill incidents and reports.
Heuristic compose
Default narrative is assembled from evidence JSON. It is precise, not lyrical. OpenAI is opt-in on top of the same contracts.
Graph is yours
Wrong edges produce wrong origins. Reject the report and fix configs/service_dependencies.yaml — do not edit facts.
Questions
Do I need an OpenAI key to use the product?
No. The console runs the full pipeline with grounded heuristic reports. Groq or OpenAI can rewrite the same evidence contracts; they are not the analysis engine.
Where does the data live?
On disk, under artifacts/. The API never phones home. Prometheus is opt-in and host-allowlisted.
Is this a replacement for PagerDuty?
No. It is the investigation layer after the page: correlate, rank, write the handoff, keep an audit trail.
Can I bring my own logs?
Yes, as long as they sit inside the configured read allowlist (data, configs, artifacts, eval).
What happens if metrics are missing?
Degraded execution continues. The run summary records the gap. Detectors that lack support simply emit nothing instead of inventing a baseline.
How do I export a report?
CLI and API both serialize JSON, Markdown, and HTML. Approved reports can POST to a webhook after URL policy checks.
Does the console persist jobs?
The job store is in-memory for the API process. Artifacts persist on disk. Re-run a scenario after a restart to refill the board.
Why not just paste logs into ChatGPT?
Because paste has no schema, no detector, no graph, no grounding, and no artifact. You get a paragraph you cannot defend.
What is closed-book Q&A?
POST /analysis-jobs/{id}/ask answers only from that run's facts, hypothesis, and retrieved snippets. If the pack does not support the question, it refuses. That refusal is intentional.
What file formats are accepted?
Logs: CSV, JSON, JSONL. Metrics: the same, plus an optional Prometheus query_range adapter. Rows that fail schema validation are counted in the run summary.
How large a file can I send?
The local workstation path is the limit. There is no cloud upload. Keep runs inside the read allowlist so the API can open them.
Does correlation invent services?
No. Services come from ingested rows and the dependency graph. Compose is scored against unexpected service mentions in evaluation.
What is root-cause support?
Top candidate score divided by the sum of all candidate scores. It ranks origins inside one incident. It is not a calibrated probability.
Can I run stages one at a time?
Yes. The CLI exposes ingest, normalize, detect, correlate, and RCA as separate commands. The console always runs the full compiler.