'Openshift 4.8 fluentd causing selinux AVC denials on the host server in rsyslog
Not sure what I am missing, but I have an OCP 4.8 cluster deployed on my host machine where the openshift nodes are VMs running on the host machine. I have the the logging and elasticsearch operators installed and I have successfully deployed the Namespaces, OperatorGroups, Subscriptions and ClusterLogging CRs. The ClusterLogging and ClusterLogForwarder CR are provided below and I can provide the other resources if needed. All of the pods are up and running and viewing each of the pods logs shows no issues or errors:
NAME READY STATUS RESTARTS AGE
cluster-logging-operator-5bd59d6cf6-r7ghg 1/1 Running 6 4d1h
elasticsearch-cdm-eli9eteu-1-78bf4d554-6gnrr 2/2 Running 0 27m
elasticsearch-cdm-eli9eteu-2-74cb5b7dd8-2k5xp 2/2 Running 0 27m
elasticsearch-cdm-eli9eteu-3-85f5dcbd66-hp8mv 2/2 Running 0 27m
elasticsearch-im-app-27434550-cq7mg 0/1 Completed 0 74s
elasticsearch-im-audit-27434550-l2f2f 0/1 Completed 0 74s
elasticsearch-im-infra-27434550-qqls5 0/1 Completed 0 74s
fluentd-7mw72 1/1 Running 0 27m
fluentd-d7nmb 1/1 Running 0 27m
fluentd-f76cb 1/1 Running 0 27m
fluentd-gj5lb 1/1 Running 0 27m
fluentd-p6228 1/1 Running 0 27m
fluentd-s7k8n 1/1 Running 0 27m
kibana-559dff8d4d-bwfpd 2/2 Running 0 27m
I've created a Kibana index and I can see the Openshift logs as I would expect in Kibana; however when I deploy the ClusterLogForwarder resource below and I have the UDP or TCP modules loaded in rsyslog I am receiving constant AVC denials for curl requests in rsyslog and a lot of other fluentd logs like below, but I am not receiving the openshift logs:
682F2F7365637265742F61646D696E2D63657274002D2D6B6579002F6574632F656C61737469637365617263682F2F7365637265742F61646D696E2D6B6579002D4800436F6E74656E742D747970653A6170706C69636174696F6E2F6A
Mar 1 13:28:53 paasmaster2.ocp4.compute.internal fluentd: I0301 13:28:52.141284 1 certsync_controller.go:66] Syncing configmaps: []
Mar 1 13:28:53 paasmaster2.ocp4.compute.internal fluentd: I0301 13:28:52.141412 1 certsync_controller.go:170] Syncing secrets: [{kube-scheduler-client-cert-key false}]
Mar 1 13:28:53 paasmaster2.ocp4.compute.internal fluentd: I0301 13:28:51.610056 22 apiaccess_count_controller.go:147] finished updating top monitoring.coreos.com/v1, Resource=thanosrulers APIRequest counts
Mar 1 13:28:53 paasmaster2.ocp4.compute.internal fluentd: W0301 13:28:51.999522 22 admission.go:98] ClusterQuotaAdmission received non-meta object *unstructured.Unstructured
Mar 1 13:28:53 paasmaster2.ocp4.compute.internal fluentd: W0301 13:28:52.015378 22 admission.go:98] ClusterQuotaAdmission received non-meta object *unstructured.Unstructured
Mar 1 13:28:53 paasmaster2.ocp4.compute.internal fluentd: W0301 13:28:52.033009 22 admission.go:98] ClusterQuotaAdmission received non-meta object *unstructured.Unstructured
Mar 1 13:28:53 paasmaster2.ocp4.compute.internal fluentd: I0301 13:28:52.216631 22 apiaccess_count_controller.go:130] updating top policy/v1beta1, Resource=podsecuritypolicies APIRequest counts
Mar 1 13:28:53 paasmaster2.ocp4.compute.internal fluentd: I0301 13:28:52.247287 22 apiaccess_count_controller.go:147] finished updating top policy/v1beta1, Resource=podsecuritypolicies APIRequest counts
Mar 1 13:28:54 paasmaster1.ocp4.compute.internal fluentd: I0301 13:28:53.182444 1 certsync_controller.go:66] Syncing configmaps: []
Mar 1 13:28:54 paasmaster1.ocp4.compute.internal fluentd: I0301 13:28:53.182464 1 certsync_controller.go:170] Syncing secrets: [{kube-scheduler-client-cert-key false}]
Mar 1 13:28:54 paasmaster1.ocp4.compute.internal fluentd: time="2022-03-01T13:28:53Z" level=info msg="[status] Previous and current ClusterOperator Status are the same, the ClusterOperator Status will not be updated."
Mar 1 13:28:53 paasworker1.ocp4.compute.internal fluentd: type=AVC msg=audit(1646141332.108:6700): avc: denied { module_request } for pid=660184 comm="curl" kmod="net-pf-10" scontext=system_u:system_r:container_t:s0:c4,c27 tcontext=system_u:system_r:kernel_t:s0 tclass=system permissive=0
Mar 1 13:28:53 paasworker1.ocp4.compute.internal fluentd: type=SYSCALL msg=audit(1646141332.108:6700): arch=c000003e syscall=41 success=no exit=-97 a0=a a1=2 a2=0 a3=12 items=0 ppid=660158 pid=660184 auid=4294967295 uid=1000710000 gid=0 euid=1000710000 suid=1000710000 fsuid=1000710000 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="curl" exe="/usr/bin/curl" subj=system_u:system_r:container_t:s0:c4,c27 key=(null)#035ARCH=x86_64 SYSCALL=socket AUID="unset" UID="unknown(1000710000)" GID="root" EUID="unknown(1000710000)" SUID="unknown(1000710000)" FSUID="unknown(1000710000)" EGID="root" SGID="root" FSGID="root"
Mar 1 13:28:53 paasworker1.ocp4.compute.internal fluentd: type=PROCTITLE msg=audit(1646141332.108:6700): proctitle=6375726C002D73002D2D68656164002D2D636163657274002F6574632F656C61737469637365617263682F2F7365637265742F61646D696E2D6361002D2D63657274002F6574632F656C61737469637365617263682F2F7365637265742F61646D696E2D63657274002D2D6B6579002F6574632F656C61737469637365617263
It appears to be an SELinux issue and I have tried setting SELinux to Permissive on the host for testing, but I'm still getting the denials. When I do deploy the ClusterLogForwarder resource I can view the Openshift logs in Kibina. Also, verified fluentd pods are running with a priviliged SCC context.
Any ideas would be greatly appreciated.
ClusterLogging CR:
apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogging"
metadata:
name: "instance"
namespace: "openshift-logging"
spec:
managementState: "Managed"
logStore:
type: "elasticsearch"
retentionPolicy:
application:
maxAge: 1d
infra:
maxAge: 7d
audit:
maxAge: 7d
elasticsearch:
nodeCount: 3
resources:
limits:
memory: "16Gi"
requests:
memory: "16Gi"
proxy:
resources:
limits:
memory: 256Mi
requests:
memory: 256Mi
redundancyPolicy: "SingleRedundancy"
visualization:
type: "kibana"
kibana:
replicas: 1
collection:
logs:
type: "fluentd"
fluentd: {}
ClusterLogForwarder CR:
apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
name: instance
namespace: openshift-logging
spec:
outputs:
- name: rsyslog-east
type: syslog
syslog:
facility: user
rfc: RFC3164
payloadKey: message
severity: informational
url: 'udp://<URL REMOVED>:514'
pipelines:
- name: syslog-east
inputRefs:
- audit
- application
- infrastructure
outputRefs:
- rsyslog-east
- default
UPDATE:
The openshift logs get forwarded if I use the syslog legacy format of using a syslog config file with a config map; however I still receive the AVC denials too.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
