'Openshift 4.8 fluentd causing selinux AVC denials on the host server in rsyslog

Not sure what I am missing, but I have an OCP 4.8 cluster deployed on my host machine where the openshift nodes are VMs running on the host machine. I have the the logging and elasticsearch operators installed and I have successfully deployed the Namespaces, OperatorGroups, Subscriptions and ClusterLogging CRs. The ClusterLogging and ClusterLogForwarder CR are provided below and I can provide the other resources if needed. All of the pods are up and running and viewing each of the pods logs shows no issues or errors:

    NAME                                            READY   STATUS      RESTARTS   AGE
cluster-logging-operator-5bd59d6cf6-r7ghg       1/1     Running     6          4d1h
elasticsearch-cdm-eli9eteu-1-78bf4d554-6gnrr    2/2     Running     0          27m
elasticsearch-cdm-eli9eteu-2-74cb5b7dd8-2k5xp   2/2     Running     0          27m
elasticsearch-cdm-eli9eteu-3-85f5dcbd66-hp8mv   2/2     Running     0          27m
elasticsearch-im-app-27434550-cq7mg             0/1     Completed   0          74s
elasticsearch-im-audit-27434550-l2f2f           0/1     Completed   0          74s
elasticsearch-im-infra-27434550-qqls5           0/1     Completed   0          74s
fluentd-7mw72                                   1/1     Running     0          27m
fluentd-d7nmb                                   1/1     Running     0          27m
fluentd-f76cb                                   1/1     Running     0          27m
fluentd-gj5lb                                   1/1     Running     0          27m
fluentd-p6228                                   1/1     Running     0          27m
fluentd-s7k8n                                   1/1     Running     0          27m
kibana-559dff8d4d-bwfpd                         2/2     Running     0          27m

I've created a Kibana index and I can see the Openshift logs as I would expect in Kibana; however when I deploy the ClusterLogForwarder resource below and I have the UDP or TCP modules loaded in rsyslog I am receiving constant AVC denials for curl requests in rsyslog and a lot of other fluentd logs like below, but I am not receiving the openshift logs:

682F2F7365637265742F61646D696E2D63657274002D2D6B6579002F6574632F656C61737469637365617263682F2F7365637265742F61646D696E2D6B6579002D4800436F6E74656E742D747970653A6170706C69636174696F6E2F6A
Mar  1 13:28:53 paasmaster2.ocp4.compute.internal fluentd: I0301 13:28:52.141284       1 certsync_controller.go:66] Syncing configmaps: []
Mar  1 13:28:53 paasmaster2.ocp4.compute.internal fluentd: I0301 13:28:52.141412       1 certsync_controller.go:170] Syncing secrets: [{kube-scheduler-client-cert-key false}]
Mar  1 13:28:53 paasmaster2.ocp4.compute.internal fluentd: I0301 13:28:51.610056      22 apiaccess_count_controller.go:147] finished updating top monitoring.coreos.com/v1, Resource=thanosrulers APIRequest counts
Mar  1 13:28:53 paasmaster2.ocp4.compute.internal fluentd: W0301 13:28:51.999522      22 admission.go:98] ClusterQuotaAdmission received non-meta object *unstructured.Unstructured
Mar  1 13:28:53 paasmaster2.ocp4.compute.internal fluentd: W0301 13:28:52.015378      22 admission.go:98] ClusterQuotaAdmission received non-meta object *unstructured.Unstructured
Mar  1 13:28:53 paasmaster2.ocp4.compute.internal fluentd: W0301 13:28:52.033009      22 admission.go:98] ClusterQuotaAdmission received non-meta object *unstructured.Unstructured
Mar  1 13:28:53 paasmaster2.ocp4.compute.internal fluentd: I0301 13:28:52.216631      22 apiaccess_count_controller.go:130] updating top policy/v1beta1, Resource=podsecuritypolicies APIRequest counts
Mar  1 13:28:53 paasmaster2.ocp4.compute.internal fluentd: I0301 13:28:52.247287      22 apiaccess_count_controller.go:147] finished updating top policy/v1beta1, Resource=podsecuritypolicies APIRequest counts
Mar  1 13:28:54 paasmaster1.ocp4.compute.internal fluentd: I0301 13:28:53.182444       1 certsync_controller.go:66] Syncing configmaps: []
Mar  1 13:28:54 paasmaster1.ocp4.compute.internal fluentd: I0301 13:28:53.182464       1 certsync_controller.go:170] Syncing secrets: [{kube-scheduler-client-cert-key false}]
Mar  1 13:28:54 paasmaster1.ocp4.compute.internal fluentd: time="2022-03-01T13:28:53Z" level=info msg="[status] Previous and current ClusterOperator Status are the same, the ClusterOperator Status will not be updated."
Mar  1 13:28:53 paasworker1.ocp4.compute.internal fluentd: type=AVC msg=audit(1646141332.108:6700): avc:  denied  { module_request } for  pid=660184 comm="curl" kmod="net-pf-10" scontext=system_u:system_r:container_t:s0:c4,c27 tcontext=system_u:system_r:kernel_t:s0 tclass=system permissive=0
Mar  1 13:28:53 paasworker1.ocp4.compute.internal fluentd: type=SYSCALL msg=audit(1646141332.108:6700): arch=c000003e syscall=41 success=no exit=-97 a0=a a1=2 a2=0 a3=12 items=0 ppid=660158 pid=660184 auid=4294967295 uid=1000710000 gid=0 euid=1000710000 suid=1000710000 fsuid=1000710000 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="curl" exe="/usr/bin/curl" subj=system_u:system_r:container_t:s0:c4,c27 key=(null)#035ARCH=x86_64 SYSCALL=socket AUID="unset" UID="unknown(1000710000)" GID="root" EUID="unknown(1000710000)" SUID="unknown(1000710000)" FSUID="unknown(1000710000)" EGID="root" SGID="root" FSGID="root"
Mar  1 13:28:53 paasworker1.ocp4.compute.internal fluentd: type=PROCTITLE msg=audit(1646141332.108:6700): proctitle=6375726C002D73002D2D68656164002D2D636163657274002F6574632F656C61737469637365617263682F2F7365637265742F61646D696E2D6361002D2D63657274002F6574632F656C61737469637365617263682F2F7365637265742F61646D696E2D63657274002D2D6B6579002F6574632F656C61737469637365617263

It appears to be an SELinux issue and I have tried setting SELinux to Permissive on the host for testing, but I'm still getting the denials. When I do deploy the ClusterLogForwarder resource I can view the Openshift logs in Kibina. Also, verified fluentd pods are running with a priviliged SCC context.

Any ideas would be greatly appreciated.

ClusterLogging CR:

apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogging"
metadata:
  name: "instance"
  namespace: "openshift-logging"
spec:
  managementState: "Managed"
  logStore:
    type: "elasticsearch"
    retentionPolicy:
      application:
        maxAge: 1d
      infra:
        maxAge: 7d
      audit:
        maxAge: 7d
    elasticsearch:
      nodeCount: 3
      resources:
        limits:
          memory: "16Gi"
        requests:
          memory: "16Gi"
      proxy:
        resources:
          limits:
            memory: 256Mi
          requests:
            memory: 256Mi
      redundancyPolicy: "SingleRedundancy"
  visualization:
    type: "kibana"
    kibana:
      replicas: 1
  collection:
    logs:
      type: "fluentd"
      fluentd: {}

ClusterLogForwarder CR:

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  outputs:
   - name: rsyslog-east
     type: syslog
     syslog:
       facility: user
       rfc: RFC3164
       payloadKey: message
       severity: informational
     url: 'udp://<URL REMOVED>:514'
  pipelines:
   - name: syslog-east
     inputRefs:
     - audit
     - application
     - infrastructure
     outputRefs:
     - rsyslog-east
     - default

UPDATE:

The openshift logs get forwarded if I use the syslog legacy format of using a syslog config file with a config map; however I still receive the AVC denials too.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source