When the attacker mutes Grafana: your observability as evasion
A stolen Grafana service-account token does not just read dashboards. It mutes and deletes the alert rules that would catch the next move, turning your observability plane into an evasion surface.
Grafana is where you watch everything else, which makes it a prize. A stolen service-account token gives read access to dashboards and data sources, but the distinct danger is what it does to alerting.
How the attack works
The token authenticates to the API from a host that has never used it, lists all data sources, and exports dashboard JSON including connection metadata. Then it turns on the monitoring stack itself: it creates broad silences and deletes alert rules covering authentication anomalies and egress spikes, and edits contact points so remaining alerts route to a dead webhook. With detection blinded, downstream attacks proceed unseen. In ATT&CK terms this is T1562, Impair Defenses, riding on T1078, Valid Accounts.
Why it works
The token had broad org-admin scope with no source restriction, so a read-only monitoring credential could edit and delete alert rules and notification policies.
How to fix it
The principal is a token, so revoke that specific service-account token to end the session, not the admin password and not a server restart that the still-valid token can reconnect through. The recovery move the scenario teaches is that alerting is config-as-code: restore the muted and deleted rules from provisioning or IaC to bring detection back online. Scope what was silenced from the Grafana audit log filtered to the token and window, since the live UI only shows the after-state. As a class fix, scope service-account tokens to least privilege, split read-only viewer from alert-editor roles, and manage alerting as code so drift is detectable and reversible. Treat the blind window as unmonitored risk, not proof of safety.
Practice it
We built this as a GraphLattice Range scenario so responders can rehearse revoking the token, restoring alert rules from code, and scoping what was silenced from the audit log.