Random 502 Bad gateway error using Azure application Gateway with Traefik Ingress for MultiSite

Question

I have created an Azure application gateway WAF V2 tier, which is connected to Traefik ingress controller in AKS and the ingress forward traffic to the applications in AKS. The backend-pool point to Treafik ingress private IP and the Http Setting and Listener have been satup for multisite. I have also sat up HealthProbe which is tested successfully. The timeout has been increased to 180 seconds.

Sending a get requests to app host works fine for a few minutes then it starts to give 502 bad gateway response for some seconds and later on starts to give 200 ok reponse. So as you can see it is not stable.

The moment that it gives 502 error, Backend-health and HealthProbe works fine. There are not much info in logs which you can see it here:

Application Gateway (WAF) -> Traefik Ingress Controller -> services

AzureDiagnostics | where ResourceProvider == "MICROSOFT.NETWORK" and Category has "ApplicationGatewayAccessLog" and httpStatus_d == 502

TimeGenerated [UTC]
2021-10-08T08:05:21.533Z
ResourceId
/SUBSCRIPTIONS/.../RESOURCEGROUPS/..../PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/MyAPPGATEWAY
Category
ApplicationGatewayAccessLog
ResourceGroup
RG-APP-SANDBOX
SubscriptionId
......
ResourceProvider
MICROSOFT.NETWORK
Resource
MyAPPGATEWAY
ResourceType
APPLICATIONGATEWAYS
OperationName
ApplicationGatewayAccess
requestUri_s
/
userAgent_s
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0
ruleName_s
vnet-sandbox-rqrt-https
httpMethod_s
GET
instanceId_s
appgw_0
httpVersion_s
HTTP/1.1
clientIP_s
......
host_s
hellopython-aks.mydomain.io
sslEnabled_s
on
clientPort_d
52260
httpStatus_d
502
receivedBytes_d
495
sentBytes_d
366
timeTaken_d
0.004
SourceSystem
Azure
timeStamp_t [UTC]
2021-10-08T08:04:42Z
transactionId_g
.....
listenerName_s
vnet-sandbox-listener-https
backendPoolName_s
vnet-sandbox-bp
backendSettingName_s
vnet-sandbox-hts-https
originalRequestUriWithArgs_s
/
sslCipher_s
ECDHE-RSA-.....
sslProtocol_s
TLSv1.2
sslClientVerify_s
NONE
serverRouted_s
MyIngressPrivateIP:443
serverStatus_s
502
serverResponseLatency_s
0.004
originalHost_s
hellopython-aks.mydomain.io
Type
AzureDiagnostics
_ResourceId
/subscriptions/...../resourcegroups/...../providers/microsoft.network/applicationgateways/myappgateway

Does anyone know why this happens and how I can fix it?