CloudFront gives 403 when origin request policy (Include all headers & querystring) is added?

Question

I am trying to set up a CF distribution with custom origin name that points to a custom domain setup in Route53 with latency based routing (to multiple regions). This is because Regional API doesn't including user IP info (XFF header) whereas CloudFront does.

In doing so, I found a really odd behavior with CF. When I configure Cache key and origin requests, CF always give undesired output. Below is what works as I just have managed policy CachingDisabled for Cache policy.

But this has an issue as none of headers is passed to the application, so I conifgured Origin request policy and added AllViewer policy since I would need all custom headers/queryparams from client to be forwarded.

This however breaks as I get an 403 error.

{
  "message": "Forbidden"
}

I've also tried Legacy cache settings with headers set to All, but this option also gives 403. I am suspecting it is something to do with some headers appended by CloudFront that get rejected at API Gateway, but I can't configure out..

What gives?

Answer 1

When I encountered this, my first instinct was to go look for the error in the docs. This ruled out a lot of possibilities and as the comments suggest, missing headers is most likely the cause. In addition, I was using WAF so that could also cause the error.

I turned on API Gateway request logging and WAF cloudwatch logging and tried a few combinations. The main issue seemed to not be with the CachingDisabled Cache Policy, but in using it with the AllViewer Origin Request Policy. Switching to having no Origin Request Policy at all yielded an error - but it was one of my errors. Sure enough, I was able to confirm that my endpoint was getting reached, and the source of the error was a missing X-Signature header. However, by making a custom Origin Request Policy policy and adding that header manually, I was able to start getting 200 OK from my endpoint. All that with the WAF working properly too. The interesting thing here is that when defining the custom policy I only needed to specify that one header - cloudfront seemed to be forwarding many other things by default like X-Forwarded-For and Host.

Reading the docs more closely, it seems that there's a lot of implicit header-passing that goes on behind the scenes when using these policies. Headers in the cache key are sent to the origin automatically, and AllViewer is really vague. I was unable to find a list of explicit headers in the policy itself, but I suspect that it's these ones. It's behaving as if it's passing those headers and nothing else with AllViewer, possibly not even Host or x-apigw-api-id, two headers that APIG seems to need. I wasn't able to empirically confirm what's getting passed, but that's my working theory anyway.

The TL;DR here is that 1) yes, you can use CachingDisabled policy with APIG and 2) you need to be very careful to check what's actually getting passed from cloudfront to APIG. In my case it was headers but depending on your use case you might also run into issues with cookies, compression support, or the querystring.