AWS Network LoadBalancer listener and the following target groups have incompatible protocols

Question

I have run into an issue trying to setup rtmp on AWS EC2 using CloudFormation. I have two stack's aws.yaml and web.yaml. CF responds with the following error on execution: Internal error reported from downstream service during operation 'The listener and the following target groups have incompatible protocols: [arn:...]. I've tried to specify a protocol directly in the listener but this just results in the error, You cannot specify the protocol version for a target group with the 'TCP_UDP' protocol. Thanks in advance for help!

nlb.yaml

AWSTemplateFormatVersion: 2010-09-09
Description: Deploys a single network load balancer

Parameters:
  VpcId:
    Type: AWS::EC2::VPC::Id
    Description: ID of the VPC

  PublicSubnets:
    Type: List<AWS::EC2::Subnet::Id>
    Description: List of Public subnets to use for the Load Balancer

  CertificateArn:
    Type: String
    Description: Arn of the ACM certificate to be used by the load balancer listener.

Resources:
  PublicNlb:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      IpAddressType: ipv4
      Name: publicNlb
      Scheme: internet-facing
      Subnets: !Ref PublicSubnets
      Tags:
        - Key: Name
          Value: wordpress-nlb
      Type: network

  PublicNlbTargetGroupHttp:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      Port: 80
      Protocol: TCP
      Targets:
        - Id: !ImportValue WordPressPublicAlbArn
      TargetType: alb
      VpcId: !Ref VpcId

  PublicNlbTargetGroupHttps:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      Port: 443
      Protocol: TCP
      Targets:
        - Id: !ImportValue WordPressPublicAlbArn
      TargetType: alb
      VpcId: !Ref VpcId

  PublicNlbHttpListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      DefaultActions:
        - Type: forward
          TargetGroupArn: !Ref PublicNlbTargetGroupHttp
      LoadBalancerArn: !Ref PublicNlb
      Port: 80
      Protocol: TCP

  PublicNlbHttpsListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      DefaultActions:
        - Type: forward
          TargetGroupArn: !Ref PublicNlbTargetGroupHttps
      LoadBalancerArn: !Ref PublicNlb
      Port: 443
      Protocol: TCP

Outputs:
  PublicNlbTargetGroupHttpArn:
    Value: !Ref PublicNlbTargetGroupHttp
    Export:
      Name: WordPressPublicNlbTargetGroupHttpArn
  PublicNlbTargetGroupHttpsArn:
    Value: !Ref PublicNlbTargetGroupHttps
    Export:
      Name: WordPressPublicNlbTargetGroupHttpsArn
  PublicNlbCanonicalHostedZoneId:
    Value: !GetAtt PublicNlb.CanonicalHostedZoneID
  PublicNlbDnsName:
    Value: !GetAtt PublicNlb.DNSName
  PublicNlbFullName:
    Value: !GetAtt PublicNlb.LoadBalancerFullName
  PublicNlbHostname:
    Value:
      !Sub https://${PublicNlb.DNSName}
    # !If [ NoSslCertificate, !Join [ '', [ 'http://', !GetAtt PublicAlb.DNSName ] ], !Join [ '', [ 'https://', !GetAtt PublicAlb.DNSName ] ] ]
  PublicNlbHttpsListenerArn:
    Value: !Ref PublicNlbHttpsListener
    Export:
      Name: WordPressPublicNlbHttpsListenerArn
  PublicNlbHttpListenerArn:
    Value: !Ref PublicNlbHttpListener
    Export:
      Name: WordPressPublicNlbHttpListenerArn

web.yaml

AWSTemplateFormatVersion: "2010-09-09"
Description: Deploys highly available wordpress application

Parameters:
  VpcId:
    Type: AWS::EC2::VPC::Id
    Description: ID of the VPC

  RecipeVersion:
    Type: String
    Description: The semantic version number you want to give to the recipe (in Major.Minor.Patch format).
    AllowedPattern: ^[0-9]+\.[0-9]+\.[0-9]+$

  PrivateSubnets:
    Type: List<AWS::EC2::Subnet::Id>
    Description: List of Private subnets to use for the application

  PrivateSubnetsString:
    Type: String
    Description: List of Private subnets to use for the application as  string

  Account:
    Type: String
    Default: non-prod
    AllowedValues:
      - prod
      - non-prod

  Environment:
    Type: String
    Default: dev
    AllowedValues:
      - dev
      - preprod
      - prod

  InstanceProfileName:
    Type: String
    Default: EC2RoleForSSM
    Description: The name of the InstanceProfile to be used with the instances.

  AmazonLinuxAMI:
    Type: AWS::EC2::Image::Id

  MinSize:
    Type: Number
    Description: The minimum limit of allowed instances to be deployed.
    Default: 1

  MaxSize:
    Type: Number
    Description: The maximum limit of allowed instances to be deployed.
    Default: 8

  DesiredCapacity:
    Type: Number
    Description: The average amount of instances to be deployed.
    Default: 1

  OnDemandBaseCapacity:
    Type: Number
    Description: The minimum amount of the Auto Scaling group's capacity that must be fulfilled by On-Demand Instances. For production you should always have at least 1 On-Demand instance (set to 1).
    Default: 1

  OnDemandPercentageAboveBaseCapacity:
    Type: Number
    Description: Controls the percentages of On-Demand Instances and Spot Instances for your additional capacity beyond OnDemandBaseCapacity. Expressed as a number (for example, 20 specifies 20% On-Demand Instances, 80% Spot Instances). Defaults to 0 if not specified. If set to 100, only On-Demand Instances are provisioned.
    Default: 0

Mappings:
  LoadBalancerRulePriority:
    dev:
      "rule": "3"
    preprod:
      "rule": "2"
    prod:
      "rule": "1"
  LoadBalancerHosts:
    dev:
      hostOne: "*.example.*"
    preprod:
      hostOne: "preprod.example.*"
    prod:
      hostOne: "*.example.*"
  SalingServerProperties:
    # @link https://aws.amazon.com/ec2/instance-types/
    # @link https://instaguide.io/#sortField=vcpu&sortDir=-1&platform=rhel&tenancyCode=dedicated&priceCalc=abs&period=mo
    dev:
      instanceTypeOne: "c6gd.xlarge"
    preprod:
      instanceTypeOne: "c6gd.xlarge"
    prod:
      instanceTypeOne: "c6gd.xlarge"

Conditions:
  IsProd: !Equals [!Ref Environment, prod]
  IsPreProd: !Equals [!Ref Environment, preprod]
  IsDev: !Equals [!Ref Environment, dev]

Resources: 
  AlbTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      Name: !Sub "${Environment}-wordpress"
      HealthCheckEnabled: true
      HealthCheckIntervalSeconds: 10
      HealthCheckPath: /
      HealthCheckPort: "80"
      HealthCheckProtocol: HTTP
      HealthCheckTimeoutSeconds: 5
      HealthyThresholdCount: 4
      UnhealthyThresholdCount: 2
      Port: 80
      Protocol: HTTP
      ProtocolVersion: HTTP1
      Tags:
        - Key: Name
          Value: !Sub "${Environment}-${RecipeVersion}-wordpress"
      TargetGroupAttributes:
        - Key: stickiness.enabled
          Value: "false"
        - Key: stickiness.type
          Value: lb_cookie # or app_cookie
      TargetType: instance
      VpcId: !Ref VpcId

  AlbListenerRule:
    Type: AWS::ElasticLoadBalancingV2::ListenerRule
    Properties:
      Actions:
        - Type: forward
          TargetGroupArn: !Ref AlbTargetGroup
      Conditions:
        - Field: host-header
          HostHeaderConfig:
            Values:
              - dropingaming.*
              - !FindInMap [ LoadBalancerHosts, !Ref Environment, hostOne ]
      ListenerArn: !ImportValue WordPressPublicAlbHttpsListenerArn
      Priority: !FindInMap [ LoadBalancerRulePriority, !Ref Environment, rule ]

  NlbTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      Name: !Sub "${Environment}-wordpress-rtmp"
      HealthCheckEnabled: true
      HealthCheckIntervalSeconds: 30
      HealthCheckPort: "4444"
      HealthCheckProtocol: HTTP
      Port: 1935
      Protocol: TCP_UDP
      Tags:
        - Key: Name
          Value: !Sub "${Environment}-${RecipeVersion}-wordpress"
      TargetGroupAttributes:
        - Key: stickiness.enabled
          Value: "true"
      TargetType: instance
      VpcId: !Ref VpcId

  NlbListenerRule:
    Type: AWS::ElasticLoadBalancingV2::ListenerRule
    Properties:
      Actions:
        - Type: forward
          TargetGroupArn: !Ref NlbTargetGroup
      Conditions:
        # Configures the rule to apply when the host header matches the domain we want to send to the target group
        # https://stackoverflow.com/questions/45664638/how-to-make-a-list-item-conditional-in-cloud-formation-template
        - Field: host-header
          HostHeaderConfig:
            Values:
              - dropingaming.*
              - !FindInMap [ LoadBalancerHosts, !Ref Environment, hostOne ]
      ListenerArn: !ImportValue WordPressPublicAlbHttpsListenerArn
      Priority: !FindInMap [ LoadBalancerRulePriority, !Ref Environment, rule ]

  LaunchTemplate:
    Type: AWS::EC2::LaunchTemplate
    Properties:
      LaunchTemplateName: !Sub "${Environment}-wordpress"
      LaunchTemplateData:
        ImageId: !Ref AmazonLinuxAMI
        EbsOptimized: 'true'
        IamInstanceProfile:
          Name: EC2RoleForSSM
        SecurityGroupIds:
          - !ImportValue WordPressEc2SecurityGroupId
        # https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html
        # The user data to make available to the instance. You must provide base64-encoded text.
        # User data is limited to 16 KB. For more information, see Running Commands on Your Linux Instance at Launch (Linux) or Adding User Data (Windows).
        # logging is sent to :: /var/log/cloud-init-output.log
        # this script is stored to :: /var/lib/cloud/instance/scripts/part-001
        UserData:
          Fn::Base64: !Sub
            - |
              #!/bin/bash -ex

              EC2_INSTANCE_ID="`wget -q -O - http://instance-data/latest/meta-data/instance-id || die \"wget instance-id has failed: $?\"`"

              test -n "$EC2_INSTANCE_ID" || die 'cannot obtain instance-id'

              EC2_AVAIL_ZONE="`wget -q -O - http://instance-data/latest/meta-data/placement/availability-zone || die \"wget availability-zone has failed: $?\"`"

              test -n "$EC2_AVAIL_ZONE" || die 'cannot obtain availability-zone'

              EC2_REGION="`echo \"$EC2_AVAIL_ZONE\" | sed -e 's:\([0-9][0-9]*\)[a-z]*\$:\\1:'`"
              
              EC2_PUBLIC_IPV4="`curl http://checkip.amazonaws.com`"
              
              EC2_PRIVATE_IPV4="`curl http://169.254.169.254/latest/meta-data/local-ipv4`"

              # https://tldp.org/LDP/abs/html/here-docs.html
              sudo cat > '/var/ebs-www-copy/aws.json' <<End-of-message
              {
                "AmazonLinuxAMI": "${AmazonLinuxAMI}",
                "RecipeVersion": "${RecipeVersion}",
                "EfsId": "${EfsId}",
                "VpcId": "${VpcId}",
                "PrivateSubnets": "${PrivateSubnetsString}",
                "Account": "${Account}",
                "Environment": "${Environment}",
                "InstanceProfileName": "${InstanceProfileName}",
                "MinSize": "${MinSize}",
                "MaxSize": "${MaxSize}",
                "DesiredCapacity": "${DesiredCapacity}",
                "OnDemandBaseCapacity": "${OnDemandBaseCapacity}",
                "OnDemandPercentageAboveBaseCapacity": "${OnDemandPercentageAboveBaseCapacity}",
                "EC2_PUBLIC_IPV4": "$EC2_PUBLIC_IPV4",
                "EC2_PRIVATE_IPV4": "$EC2_PRIVATE_IPV4",
                "EC2_INSTANCE_ID": "$EC2_INSTANCE_ID",
                "EC2_AVAIL_ZONE": "$EC2_AVAIL_ZONE",
                "EC2_REGION": "$EC2_REGION"
              }
              End-of-message

              cat /var/ebs-www-copy/aws.json

              # redacting the overly specific :)

              systemctl start httpd

            - EfsId: !ImportValue "WordPressEfsId"

  # https://github.com/aws-samples/session-manager-without-igw/blob/main/session-manager-without-igw.yml
  AutoScalingGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      AutoScalingGroupName: !Sub "${Environment}-wordpress"
      TargetGroupARNs:
        - !Ref AlbTargetGroup
        - !Ref NlbTargetGroup
      MinSize: !Ref MinSize
      MaxSize: !Ref MaxSize
      DesiredCapacity: !Ref DesiredCapacity
      VPCZoneIdentifier: !Ref PrivateSubnets
      MixedInstancesPolicy:
        LaunchTemplate:
          LaunchTemplateSpecification:
            # Version: !Ref LaunchTemplateVersionNumber
            Version: !GetAtt
              - LaunchTemplate
              - LatestVersionNumber
            LaunchTemplateId: !Ref LaunchTemplate
          # https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-mixed-instances-groups-launch-template-overrides.html
          # @link https://aws.amazon.com/ec2/instance-types/
          # @link https://instaguide.io/#sortField=vcpu&sortDir=-1&platform=rhel&tenancyCode=dedicated&priceCalc=abs&period=mo
          Overrides:
            # I want to know if predictive load b is bs or nah
            - InstanceType: !FindInMap [SalingServerProperties, !Ref Environment, instanceTypeOne]

        InstancesDistribution:
          OnDemandBaseCapacity: !Ref OnDemandBaseCapacity
          OnDemandPercentageAboveBaseCapacity: !Ref OnDemandPercentageAboveBaseCapacity
          # https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-mixed-instances-groups.html#allocation-strategies
          SpotAllocationStrategy: lowest-price
          OnDemandAllocationStrategy: lowest-price
      Tags:
        - Key: "Name"
          PropagateAtLaunch: true
          Value: !Sub "${Environment}-wordpress"
        - Key: "Version"
          PropagateAtLaunch: true
          Value: !Sub "${RecipeVersion}"

  AutoScalingPolicy:
    Type: AWS::AutoScaling::ScalingPolicy
    Properties:
      AutoScalingGroupName: !Ref AutoScalingGroup
      PolicyType: PredictiveScaling
      PredictiveScalingConfiguration:
        MetricSpecifications:
          - TargetValue: 70
            PredefinedMetricPairSpecification:
              PredefinedMetricType: ASGCPUUtilization

So based on the error the NlbListenerRule seems to be what is failing; though, the downstream service bit has me confused. Google return near nothing, so I'm hopeful you may know!?

Answer 1

So AWS support along with the comment above lead me straight to the problem. The NlbListenerRule was using an incorrect ListenerArn: !ImportValue WordPressPublicAlbHttpsListenerArn. Also, another issues which is catchable in the provided templates above was caught by the support team "NLB work on Layer-4 of OSI Model , and we cannot create NLB Listener rule based on Host-Header Condition." Thus ListenerRule's may only be applied to HTTP based hosts.

changed in web.yaml

NlbListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      DefaultActions:
        - Type: forward
          TargetGroupArn: !Ref NlbTargetGroup
      LoadBalancerArn: !ImportValue PublicNlbLoadBalancerArn
      Port: 1935
      Protocol: TCP_UDP

added in alb.yaml

Outputs:
  PublicNlb:
    Value: !Ref PublicNlb
    Export:
      Name: PublicNlbLoadBalancerArn
  PublicNlbTargetGroupHttpArn: