API Gateway integration with S3 where binary media types are not known in advance produces corrupt files

Question

I am creating an API Gateway with various endpoints, one of which gets objects that are binary files from S3 with everything defined in Terraform. It works fine if I set the gateway up with a predefined list of MIME types, eg:

resource "aws_api_gateway_rest_api" "my_api" {
...
    binary_media_types = [ "application/vnd.openxmlformats-officedocument.wordprocessingml.document", 
       "application/msword",
       "application/pdf" ]
...
}

The problem is that I don't know ahead of time all of the MIME types that will exist in the bucket. If I try to use a generic binary MIME type like application/octet-stream I get additional characters in the response body and the file can't be opened. I've tried with a MIME type of application/* but I get an internal server error (presumably it can't be mapped).

The integration response settings have content handling set to passthrough with no header mappings or mapping templates. The method response also has just a 200 response with no header or model settings.

I've seen a few questions that talk about incorrect encoding between Lambda's Base64 string output and API Gateway but my scenario is just S3 -> API Gateway. I've looked at the below links but they don't seem to help me because they use predefined binary types:

Integrating API Gateway with AWS Services S3

API Gateway Content Encodings

API Gateway Payload Encodings

Is there a way of allowing wildcard binary types in API Gateway, and then passing through the integration Content-Type to the output? Alternatively, is there a way to use a generic MIME type such as application/octet-stream and stop API Gateway from adding additional characters?

Answer 1

Please find below an example how to serve static content like images, PDF, etc. from an S3 bucket using api gateway.

Steps to reproduce:

• setup YOUR_BUCKET
• place a test image test.png into s3://YOUR_BUCKET/test/test.png
• set up a role role/apigAwsProxyRole which is allowed to read from YOUR_BUCKET
• create a new api in api gateway
• import the below OpenApi definition and overwrite your existing api
• In api gateway click on Settings on the left menu. Make sure that after the import you still have */* in the binary content type.
• deploy the api, e.g. with stage dev
• access your static file: https://0omkxxxxkj.execute-api.us-east-1.amazonaws.com/dev/v1/files/test/test.png

Notice that the same setup also works with a private api endpoint. That way you can serve static content from S3 by making it only available through a private endpoint within your VPC.

{
  "swagger" : "2.0",
  "info" : {
    "version" : "1.0",
    "title" : "s3-static"
  },
  "host" : "0omkxxxxkj.execute-api.us-east-1.amazonaws.com",
  "basePath" : "/dev",
  "schemes" : [ "https" ],
  "paths" : {
    "/v1/files/{proxy+}" : {
      "get" : {
        "produces" : [ "application/octet", "application/json" ],
        "parameters" : [ {
          "name" : "proxy",
          "in" : "path",
          "required" : true,
          "type" : "string"
        } ],
        "responses" : {
          "200" : {
            "description" : "200 response",
            "schema" : {
              "$ref" : "#/definitions/Empty"
            },
            "headers" : {
              "Content-Type" : {
                "type" : "string"
              }
            }
          },
          "500" : {
            "description" : "500 response"
          }
        },
        "x-amazon-apigateway-integration" : {
          "credentials" : "arn:aws:iam::123456789:role/apigAwsProxyRole",
          "httpMethod" : "GET",
          "uri" : "arn:aws:apigateway:us-east-1:s3:path/YOUR_BUCKET/{proxy}",
          "responses" : {
            "2\\d{2}" : {
              "statusCode" : "200",
              "responseParameters" : {
                "method.response.header.Content-Type" : "integration.response.header.Content-Type"
              }
            }
          },
          "requestParameters" : {
            "integration.request.path.proxy" : "method.request.path.proxy"
          },
          "passthroughBehavior" : "when_no_match",
          "type" : "aws"
        }
      }
    }
  },
  "definitions" : {
    "Empty" : {
      "type" : "object",
      "title" : "Empty Schema"
    }
  },
  "x-amazon-apigateway-binary-media-types" : [ "*/*" ]
}