‹ #incidents 12 members
E
Elena 8:02 AM
@Dash give me the debrief on last night's auth outage — I'm writing the postmortem
Dash App 8:05 AM
Here's the full debrief assembled from the incident channel, support queue, and deployment logs.
Incident summary — Jun 26 auth outage
Duration: 94 minutes (11:43 PM – 1:17 AM) · Severity: P1 · Customers affected: ~3% of active sessions · Data loss: none
Timeline
- 11:43 PM — First alert: elevated 401 error rate on
/auth/token. PagerDuty fires. - 11:51 PM — On-call engineer (Neil) picks up and begins investigation. Identifies that the 8:30 PM deploy introduced a token validation change.
- 12:04 AM — Root cause confirmed: a new env variable for token expiry wasn't set in production, causing session tokens to validate as expired on first refresh.
- 12:11 AM — Rollback prepared. Neil can't self-merge — needs a second engineer to approve. Attempts to reach backup on-call; 18-minute response gap.
- 12:29 AM — Second engineer approves rollback. Deploy begins.
- 12:47 AM — Rollback live. Error rate returns to baseline. Hotfix with the correct env var value prepared in parallel.
- 1:05 AM — Hotfix shipped with the env var fix and a guard to prevent the same class of misconfiguration.
- 1:17 AM — Incident closed. 12 support tickets opened; all contacted within 2 hours.
Root cause
A token expiry env variable was added to the codebase but not set in the production environment. The deploy-time checklist didn't cover env var validation for auth-related changes. The rollback was delayed 18 minutes by the self-merge policy — the on-call engineer needed approval but couldn't reach backup immediately.
Action items for the postmortem
- Add a pre-deploy env var validation step for any change touching the auth service — blocks deploy if required vars are unset
- Update the on-call runbook to define a Sev-1 break-glass self-merge policy with a 10-minute timeout before it applies
- Add token validation success rate to the post-deploy smoke test suite
- Close all 12 support tickets by EOD today with a written explanation and a goodwill credit offer
🙏 4 💯 2
+ Message #incidents ☺