How does target response time aws metric work

Question

I have a tomcat application deployed on ec2 and behind an application load balancer. The load balancer also has an alarm set for target response time which looks like this(screenshot), target response time threshold

Now the metric says response time cannot be greater than 1ms. But when I look at the chart there are multiple instances of the target response times going over 2000ms for continuous 5 minutes as shown in pic, response time chart

And yet, the alarm only goes off sometimes and not all the time.

1. Why is the alarm not going off almost all the time?
2. Can someone better explain target response time than the explanation given at https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-cloudwatch-metrics.html
3. Is 1ms a realistic target response time to measure if I see sometimes the values going to 5000ms?
4. I see the message in pic 1 "Value will be converted to match cloudwatch metric units". And yet in cloudwatch, the alarm generated is of unhealthy host with no mapping to the 1ms value

Answer 1

I stumbled onto this from searching around for my own problems. I know this is old, but I thought I should post some info for others that come across this.

If you follow through to the ELB access log definition of target_processing_time:

The total time elapsed (in seconds, with millisecond precision) 
from the time the load balancer sent the request to a target 
until the target started to send the response headers.

This is a pretty dumb statistic for Amazon to offer, and isn't really that useful. If your web app is unbuffered and spits out response headers right away, this value will be consistently low and have little bearing on your real performance.

A better method is to track your own delivery times within your app and publish custom metrics for them.