On Saturday, April 26, alerts began firing in the Grafana Labs Security Department. One of the thousands of canary tokens we have deployed across our code and infrastructure had been triggered, immediately notifying our global team.
Our security team quickly mobilized to investigate the alerts, assess the impact, and initiate mitigation efforts. After reviewing recent changes, we identified the root cause: a recently enabled GitHub Action allowed an unauthorized user access to a limited number of tokens, all of which have now been invalidated.
While our investigation is ongoing, at this time, we have found no evidence of code modifications, unauthorized access to production systems, exposure of customer data, or access to personal information.
Summary & impact
By forking a Grafana repository, running a curl command to inject malicious code, and dumping environment variables to a file encrypted with a private key, the attacker was able to extract tokens. They then deleted their fork to conceal the activity. Using a compromised credential, the attacker replicated the attack against four private repositories.
The unauthorized access was limited to automation systems and did not impact production environments or release artifacts.
The attacker’s objective appeared to be to harvest tokens and remain undetected for future use; a tactic aligned with findings from a recent Mandiant report, which states the average time from credential theft to exploitation is 11 days.
Grafana Labs response
We took the following steps to mitigate the vulnerability:
- We immediately removed the vulnerable GitHub Action and disabled all workflows across public repositories.
- We rotated all exposed tokens.
- We used manual checks, combined with Trufflehog, to verify all credentials were fully invalidated.
- We audited our internal workflows using Gato-X to ensure similar vulnerabilities were not present elsewhere.
In parallel, we have launched a full audit of access logs, which we maintain in Grafana Loki. Our investigation is ongoing, but we have high confidence that the intrusion was contained to the five affected repositories.
As part of our standard security practices, we will share additional information from our post-incident review when our investigations are complete. In addition, we are strengthening our CI/CD security measures, including making tools like Zizmor and Trufflehog mandatory components of our development process. We’re proud of the early detection capabilities within our CI/CD pipeline, but we recognize there’s always room to improve.
Reporting security issues
If you think you have found a security vulnerability, please go to our Report a security issue page to learn how to send a security report.
Grafana Labs will send you a response indicating the next steps in handling your report. After the initial reply to your report, the security team will keep you informed of the progress towards a fix and full announcement, and may ask for additional information or guidance.
Important: We ask you to not disclose the vulnerability before it has been fixed and announced, unless you received a response from the Grafana Labs security team that you can do so.
Security announcements
We maintain a security category on our blog, where we will always post a summary, remediation, and mitigation details for any patch containing security fixes. You can also subscribe to our RSS feed.