Your CI pipeline is a product with users (engineers), SLAs (feedback time), and bugs (flakes). Managing it that way is cheaper than living with it as accidental infrastructure.
Users, SLAs, and bugs
Engineers are the users. Their primary metric is time-to-trustworthy-feedback. Flaky tests are bugs. Queue time is capacity planning. Silent skips are outages.
Publish a simple dashboard: median and p95 duration, flake rate, pass rate, and cache hit rate. Review it monthly with the same seriousness as service latency.
Spend speed where it compounds
Caching dependencies, parallelizing independent jobs, and running the fast checks first (lint, typecheck, unit) give the best return. Full end-to-end suites rarely need to block every push.
Bisect slow jobs with timing data. "CI is slow" is not a backlog item; "job X is 40% of p95 wall time" is.
Treat flakes as P1 product bugs
A flaky test trains people to retry instead of fix. Quarantine with a visible ticket and an owner. Track time-to-quarantine and time-to-fix.
Never permanently delete a flaky test without recording what risk it was covering.
Every ignored flake teaches the team that red builds are optional.
Own the pipeline in the open
Pipeline config lives in the repo with code owners. Changes ship through PRs. Secrets in CI are injected, not printed.
On-call for "merge is broken" should be as real as on-call for production — it is your delivery system.
Key takeaways
- Measure time-to-trustworthy-feedback as the core CI SLA.
- Optimize caching, parallelism, and check ordering first.
- Quarantine flakes with owners; do not delete silently.
- Pipeline config is code; review and own it.
- Delivery system outages deserve real on-call attention.