filebeat Failed to list *v1.Pod: Unauthorized - permmision issue

Question

i am trying to deploy a filebeat deamonset on my aks cluster

i want it to run on every node and collect all the logs generated by the pods to do so i have 5 steps

1.create user 2.create role with appropriate permissions 3.bind them 4.create config map 5.create deamonset utilizing the config map

everything was created just fine.

however upon inspection of the filebeat logs i see the following messages indicating filebeat does not have permission to list pods:

E0519 16:19:18.243183       1 reflector.go:125] github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:235: Failed to list *v1.Pod: Unauthorized
E0519 16:19:19.251644       1 reflector.go:125] github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:235: Failed to list *v1.Pod: Unauthorized

this is my yml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: default
  labels:
    k8s-app: filebeat
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat
  namespace: default
  labels:
    k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat
  namespace: default
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: default
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-

    filebeat.inputs:
    - type: container
      enabled: true
      paths:
        - /var/log/containers/*.log
      # If you setup helm for your cluster and want to investigate its logs, comment out this section.
      exclude_files: ['tiller-deploy-*']

      # To be used by Logstash for distinguishing index names while writing to elasticsearch.
      fields_under_root: true
      fields:
        index_prefix: k8s-logs

      # Enrich events with k8s, cloud metadata 
      processors:
        - add_cloud_metadata:
        - add_host_metadata:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"
    # Send events to Logstash.
    output.logstash:
      enabled: true
      hosts: ["logstash-logstash-headless.elk-stack:9600"]

    # You can set logging.level to debug to see the generated events by the running filebeat instance.
    logging.level: info
    logging.to_files: false
    logging.files:
      path: /var/log/filebeat
      name: filebeat
      keepfiles: 7
      permissions: 0644
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
spec:  
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      # Refers to our previously defined ServiceAccount.
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:7.5.0
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
          # If using Red Hat OpenShift uncomment this:
          #privileged: true
        resources:       # comment out for using full speed 
          limits:
            memory: 200Mi
          requests:
            cpu: 500m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      # Bind previously defined ConfigMap
      - name: config
        configMap:
          defaultMode: 0600
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: varlog
        hostPath:
          path: /var/log
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate

any idea what might be the problem?

Answer 1

There is no az command to directly add preauthorized clients to a app registration instead you will have to use Graph API (beta) to update the same from Graph Explorer or az rest command.

Get OauthPermissionId with az command :

az ad app show --id $appId --query "oauth2Permissions[].id"

I tested the same from Graph Explorer :

Ran Patch : https://graph.microsoft.com/beta/applications/<appObjectId>

With Request body as :

{
    "api": {
        "preAuthorizedApplications": [
            {
                "appId": "authorizedappClientID",
                "permissionIds": [
                    "oauth2PermissionId"
                ]
            }
        ]
    }
}

Output:

Reference for az rest can be fount in this SO thread answered by Joy Wang .