skyfleetdocs

Grafana

Every firing Grafana alert, from Loki logs or any other data source, gets an agent that finds the cause.

Go to Connections and press Add on Grafana under Alerts and incidents. Paste your Grafana address (for example https://acme.grafana.net) and a Grafana service account token, and press Connect. Make the token in Grafana under Administration → Users and access → Service accounts: add one with the Admin role, then Add service account token. Grafana Cloud and self-hosted Grafana work the same way, as long as Skyfleet can reach the address over HTTPS on a public address. This adds the Incident response use case to your Dashboard.

Skyfleet then makes the Grafana side and lists it:

  • A Skyfleet webhook contact point carrying this source's token.
  • A first notification policy that sends every alert to Skyfleet and continues. Your own policies move, unchanged, under a default policy right after it, so every alert still reaches the contact points it reached before.
  • A list of the alert rules that pick a contact point themselves (Configure notifications → Select contact point). Those skip notification policies, so pick Skyfleet on them, or switch them to Use notification policy, to send them here too.

Deleting the source in Skyfleet deletes the contact point and puts your policy tree back as it was.

A log alert on Loki is a rule on a LogQL query, for example sum by (service_name) (count_over_time({service_name="checkout", level="error"}[5m])) > 0. Add a summary and description to the rule: they become the start of the issue.

Skyfleet reads Grafana's own webhook body: the rule name, its summary and description, each firing series' labels and values, and the link to the rule. Each alert rule is one issue in your Inbox. A firing alert starts an agent; repeats only add to its count. Resolved alerts and Grafana's Test notification start nothing.