ISTIO External Auth : '503 upstream connect error or disconnect/reset before headers. reset reason: connection terminationroot' when access over HTTPS

Question

ISTIO version: 1.9.4 EKS Cluster version: 1.14

We have deployed ISTIO APP mesh in our project. We have deployed External Authorization using istio's documentation i.e. https://istio.io/latest/docs/tasks/security/authorization/authz-custom/.

External authorizer used (as mentioned in above documentation) : https://raw.githubusercontent.com/istio/istio/release-1.9/samples/extauthz/ext-authz.yaml

When we access any API from going into pod of another API (i.e. over http), using curl command, all works fine. External auth service gets call and all the headers are passed into external authorizer's v3 check method. Below information is passed source, principal, destination, headers: authority, method, path, accept, content-length, user-agent, x-b3-sampled, x-b3-spanid, x-b3-traceid, x-envoy-attempt-count, x-ext-authz, x-forwarded-client-certx-forwarded-proto, x-request-id.

But when we try to access the same service over https using postman, browser or from going into pod of another API and using curl with https endpoint, we get denied response from external authorizer's v3 check method. Also when we check the logs of external authorizer's v3 check method no headers are passed to it in this case.

Below is setup

Name spaces with ISTIO ejection enable : foo

1. ISTIO Config map changes

data:
  mesh: |-
    # Add the following content to define the external authorizers.
    extensionProviders:
    - name: "sample-ext-authz-grpc"
      envoyExtAuthzGrpc:
        service: "ext-authz.foo.svc.cluster.local"
        port: "9000"
    - name: "sample-ext-authz-http"
      envoyExtAuthzHttp:
        service: "ext-authz.foo.svc.cluster.local"
        port: "8000"
        includeHeadersInCheck: ["x-ext-authz"]
                    

2. External Authorizer

apiVersion: v1
kind: Service
metadata:
  name: ext-authz
  namespace: foo 
  labels:
    app: ext-authz
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 8000
  - name: grpc
    port: 9000
    targetPort: 9000
  selector:
    app: ext-authz
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ext-authz
  namespace: foo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ext-authz
  template:
    metadata:
      labels:
        app: ext-authz
    spec:
      containers:
      - image: docker.io/istio/ext-authz:0.6
        imagePullPolicy: IfNotPresent
        name: ext-authz
        ports:
        - containerPort: 8000
        - containerPort: 9000 

  
                    

3. Enable the external authorization Config

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: ext-authz
  namespace: foo
spec:
  selector:
    matchLabels:
      app: user-api
  action: CUSTOM
  provider:
    name: sample-ext-authz-grpc
  rules:
  - to:
    - operation:
        paths: ["/user/api/*"]
                    

4. PeerAuth Chagnes

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: mtlsauth
  namespace: foo
spec:
  mtls:
    mode: STRICT

5. Destination Rule

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: default
  namespace: foo
spec:
  host: "*.samplehost.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

6. Virtual Service File

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: sample-gateway
  namespace: foo
spec: 
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "sample.com"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: user-api
  namespace: foo
spec: 
  hosts:
  - "sample.com"
  gateways:
  - sample-gateway
  http:
  - match: 
    - uri:   
        prefix: /user/api/ 
    route:
    - destination:
        host: user-api
        port:
          number: 9500                 

Logs from ingress gateway:

2021-07-08T11:13:33.554104Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, connection error: desc = "transport: Error while dialing dial tcp 172.20.0.51:15012: connect: connection refused"
2021-07-08T11:13:35.420052Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2021-07-08T11:43:24.012961Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0

Answer 1

I am not sure if you are facing the issue but if seems like you have enforced mtls . Thats why in the following config for gateway. You might need to open HTTPS also apiVersion: networking.istio.io/v1alpha3 kind: Gateway metadata: name: sample-gateway namespace: foo spec: selector: istio: ingressgateway servers:

• port: number: 80 name: http protocol: HTTP hosts:
  • "sample.com"
• port: number: 443 name: https protocol: HTTPS hosts:
  • "sample.com"