k8s事件通知

起因

公司一个项目(pod)，无报错日志，健康检查也是可以过的。但是却被携程下线了。自己手动调用项目也是正常 于是开发看堆栈，最后发现问题是线上打印日志打印到线下的ELK,线下的ELK被日志写满,但是报告是写的是ELK 有问题却没有及时发现，日了藏獒了，所以决定加一个事件事件，其实很不想加，因为如果加上去一堆通知， 就好某开发写的java只要一查询一个订单列表mysql的cpu%90以上，然后每次短信通知几十条，关键也不去处理。您说 要是关了的话又怕到时候mysql跑死通知不到，不关又好烦。最后还是加上事件通知，我询问产品经理被下线的时候 能否调用他们接口通知，被告知没有这样的接口（我觉得吧没有这样的接口显然不可能）

安装kube-event

github仓库

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    name: kube-eventer
  name: kube-eventer
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kube-eventer
  template:
    metadata:
      labels:
        app: kube-eventer
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      dnsPolicy: ClusterFirstWithHostNet
      serviceAccount: kube-eventer
      containers:
        - image: registry.aliyuncs.com/acs/kube-eventer-amd64:v1.2.0-484d9cd-aliyun
          name: kube-eventer
          command:
            - "/kube-eventer"
            - "--source=kubernetes:https://kubernetes.default"
            ## .e.g,dingtalk sink demo
            - --sink=elasticsearch:http://192.168.1.1:9200?sniff=false&ver=6
            - --sink=dingtalk:https://oapi.dingtalk.com/robot/send?access_token=${TOKEN}&label=${通知标题}&level=Warning
            - --sink=webhook:https://hooks.slack.com/services/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?&level=Warning&kinds=Pod&header=Content-Type=application/json&custom_body_configmap=custom-body&custom_body_configmap_namespace=kube-system&method=POST
            - --sink=wechat:https://wechat?corp_id=wx56adfb85d5e007e6&corp_secret=${SECRET}&agent_id=1000004&to_user=&label=mvm-aliyun-k8s&level=Normal
          env:
            # If TZ is assigned, set the TZ value as the time zone
            - name: TZ
              value: "Asia/Shanghai"
          volumeMounts:
            - name: localtime
              mountPath: /etc/localtime
              readOnly: true
            - name: zoneinfo
              mountPath: /usr/share/zoneinfo
              readOnly: true
          resources:
            requests:
              cpu: 100m
              memory: 100Mi
            limits:
              cpu: 500m
              memory: 250Mi
      volumes:
        - name: localtime
          hostPath:
            path: /etc/localtime
        - name: zoneinfo
          hostPath:
            path: /usr/share/zoneinfo
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kube-eventer
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - events
    verbs:
      - get
      - list
      - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kube-eventer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-eventer
subjects:
  - kind: ServiceAccount
    name: kube-eventer
    namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-eventer
  namespace: kube-system
---
apiVersion: v1
data:
  content: '{
    "channel": "testing",
    "icon_emoji": ":k8s:",
    "username": "eventer",
    "attachments": [
        {
            "color": "warning",
            "text": "*Type*: `{{.Type}}`\n*Namespace*: `{{.InvolvedObject.Namespace}}`\n*Object*: `{{ .InvolvedObject.Kind }}/{{ .InvolvedObject.Name }}`\n*Reason*: `{{ .Reason }}`\n*Meaasge*: `{{ .Message }}`\n*Time*: `{{ .LastTimestamp }}`"
        }
    ]
  }'
kind: ConfigMap
metadata:
  name: custom-body
  namespace: kube-system

标注

kubectl apply -f kube-event.yaml
# 输出到ES
--sink=elasticsearch:http://192.168.1.73:9200?sniff=false&ver=6
# 通知到微信
--sink=wechat:https://wechat?corp_id=${id}&corp_secret=${SECRET}&agent_id=1000004&to_user=&label=&level=Normal

ES时间图 kube-event-es