Alerts of the Prometheus Alertmanager with MS Teams
Prometheus alerting through Alertmanager can be configured to send messages to Email, SMS or messangers. My company uses Microsoft Teams as team messanger. Alertmanager supports Email, HipChat, PagerDuty, Slack etc. natively except Microsoft Teams. So it needs to be used Webhook to send a third-party tool that sends messages to Microsoft Teams.
This post assumes that you are using the Prometheus operator for monitoring your Kubernetes cluster. We’re going to know how to send Prometheus alerts to Microsoft Teams.
Get the Incoming Webhook URL from Microsoft Teams
This post assumes that you are familiar with Microsoft Teams basics. Let’s start to configure Incoming Webhook.
Go to the channel, where you want to receive alerts. Click on the ... on the right side of the channel name and select Connectors from the dropdown list.
Select the Incoming Webhook connector from the list of available connectors.
Enter a name to identify this webhook later. You can also add an image which will be visible whenever a message is posted using this webhook. Click Create.
This will generate a webhook URL which can then be used to post messages to this channel. Copy this webhook and save it for later.
Configure Alertmanager of Prometheus operator
To send the third-party tool for Microsoft Teams, generate alertmanager.yaml to configure Alertmanager.
global:
resolve_timeout: 5m
receivers:
- name: prometheus-msteams
webhook_configs:
- url: "http://prometheus-msteams:2000/alertmanager"
send_resolved: true
route:
group_by:
- job
group_interval: 5m
group_wait: 30s
receiver: prometheus-msteams
repeat_interval: 12h
routes:
- match:
alertname: Watchdog
receiver: prometheus-msteams
Above file uses webhook as receiver and specifies name as prometheus-msteams. Then, encode the alertmanager.yaml using base64.
cat alertmanager.yaml | base64
Z2xvYmFsOgogIHJlc29sdmVfdGltZW91dDogNW0KcmVjZWl2ZXJzOgotIG5hbWU6IHByb21ldGhldXMtbXN0ZWFtcwogIHdlYmhvb2tfY29uZmlnczoKICAtIHVybDogImh0dHA6Ly9wcm9tZXRoZXVzLW1zdGVhbXM6MjAwMC9hbGVydG1hbmFnZXIiCiAgICBzZW5kX3Jlc29sdmVkOiB0cnVlCnJvdXRlOgogIGdyb3VwX2J5OgogIC0gam9iCiAgZ3JvdXBfaW50ZXJ2YWw6IDVtCiAgZ3JvdXBfd2FpdDogMzBzCiAgcmVjZWl2ZXI6IHByb21ldGhldXMtbXN0ZWFtcwogIHJlcGVhdF9pbnRlcnZhbDogMTJoCiAgcm91dGVzOgogIC0gbWF0Y2g6CiAgICAgIGFsZXJ0bmFtZTogV2F0Y2hkb2cKICAgIHJlY2VpdmVyOiBwcm9tZXRoZXVzLW1zdGVhbXMK
Create a Secret object with the string generated above that is used by the Prometheus operator’s Alertmanager and apply it to Kubernetes.
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
name: alertmanager-main
namespace: monitoring
type: Opaque
data:
alertmanager.yaml: Z2xvYmFsOgogIHJlc29sdmVfdGltZW91dDogNW0KcmVjZWl2ZXJzOgotIG5hbWU6IHByb21ldGhldXMtbXN0ZWFtcwogIHdlYmhvb2tfY29uZmlnczoKICAtIHVybDogImh0dHA6Ly9wcm9tZXRoZXVzLW1zdGVhbXM6MjAwMC9hbGVydG1hbmFnZXIiCiAgICBzZW5kX3Jlc29sdmVkOiB0cnVlCnJvdXRlOgogIGdyb3VwX2J5OgogIC0gam9iCiAgZ3JvdXBfaW50ZXJ2YWw6IDVtCiAgZ3JvdXBfd2FpdDogMzBzCiAgcmVjZWl2ZXI6IHByb21ldGhldXMtbXN0ZWFtcwogIHJlcGVhdF9pbnRlcnZhbDogMTJoCiAgcm91dGVzOgogIC0gbWF0Y2g6CiAgICAgIGFsZXJ0bmFtZTogV2F0Y2hkb2cKICAgIHJlY2VpdmVyOiBwcm9tZXRoZXVzLW1zdGVhbXMK
EOF
Now Alertmanager is ready for sending alerts using webhook.
Set externalLabels for multiple clusters
If you have multiple Kubernetes cluster, you need to be able to idenfy which cluster is generating alerts. Add cluster: <CLUSTER_NAME> in externalLabels to Prometheus CRD so that alerts have information of the cluster.
cat <<EOF | kubectl apply -f -
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
labels:
prometheus: k8s
name: k8s
namespace: monitoring
spec:
alerting:
alertmanagers:
- name: alertmanager-main
namespace: monitoring
port: web
baseImage: quay.io/prometheus/prometheus
nodeSelector:
beta.kubernetes.io/os: linux
replicas: 2
resources:
requests:
memory: 400Mi
ruleSelector:
matchLabels:
prometheus: k8s
role: alert-rules
securityContext:
fsGroup: 2000
runAsNonRoot: true
runAsUser: 1000
serviceAccountName: prometheus-k8s
serviceMonitorNamespaceSelector: {}
serviceMonitorSelector: {}
version: v2.7.2
externalLabels:
cluster: test
Set up the prometheus-msteams
Now, we’ll set up prometheus-msteams that send alerts generated by Prometheus to Microsoft Teams.
Download the chart
Clone this repository.
git clone https://github.com/bzon/prometheus-msteams
cd prometheus-msteams/chart
Modify the default Microsoft Teams Message card template
Modify the default Microsoft Teams Message card template like this:
vim prometheus-msteams/card.tmpl
{{ define "teams.card" }}
{
"@type": "MessageCard",
"@context": "http://schema.org/extensions",
"themeColor": "{{- if eq .Status "resolved" -}}2DC72D
{{- else if eq .Status "firing" -}}
{{- if eq .CommonLabels.severity "critical" -}}8C1A1A
{{- else if eq .CommonLabels.severity "warning" -}}FFA500
{{- else -}}808080{{- end -}}
{{- else -}}808080{{- end -}}",
"summary": "{{- if eq .CommonAnnotations.summary "" -}}
{{- if eq .CommonAnnotations.message "" -}}
{{- js .CommonLabels.cluster | reReplaceAll "_" " " | reReplaceAll `\\'` "'" -}}
{{- else -}}
{{- js .CommonAnnotations.message | reReplaceAll "_" " " | reReplaceAll `\\'` "'" -}}
{{- end -}}
{{- else -}}
{{- js .CommonAnnotations.summary | reReplaceAll "_" " " | reReplaceAll `\\'` "'" -}}
{{- end -}}",
"title": "Prometheus Alert ({{ .Status }})",
"sections": [ {{$externalUrl := .ExternalURL}}
{{- range $index, $alert := .Alerts }}{{- if $index }},{{- end }}
{
"activityTitle": "[{{ js $alert.Annotations.description | reReplaceAll "_" " " | reReplaceAll `\\'` "'" }}]({{ $externalUrl }})",
"facts": [
{{- range $key, $value := $alert.Annotations }}
{
"name": "{{ $key }}",
"value": "{{ js $value | reReplaceAll "_" " " | reReplaceAll `\\'` "'" }}"
},
{{- end -}}
{{$c := counter}}{{ range $key, $value := $alert.Labels }}{{if call $c}},{{ end }}
{
"name": "{{ $key }}",
"value": "{{ js $value | reReplaceAll "_" " " | reReplaceAll `\\'` "'" }}"
}
{{- end }}
],
"markdown": true
}
{{- end }}
]
}
{{ end }}
Deploy prometheus-msteams to Kubernetes using “helm template” command
Generate Kubernetes manifest with Helm passing “Incoming Webhook URL” of Microsoft Teams. And deploy it to Kubernetes.
helm template prometheus-msteams/ --name prometheus-msteams --set connectors.alertmanager=https://outlook.office.com/webhook/xxxx/xxxx | kubectl apply -f -
configmap/prometheus-msteams-config created
configmap/prometheus-msteams-card-template created
service/prometheus-msteams created
deployment.apps/prometheus-msteams created
Check received alerts in Microsoft Teams
You finished all configurations. Now you can receive Prometheus alerts through Microsoft Teams like this.
