S3, Signed-URLs and Caching

Question

I am generating signed urls on my webapp (nodejs) using the knox nodejs-library. However the issue arises, that for every request, I need to regenerate an unique GET signed url for the current user, leaving browser's cache-control out of the game.

I've searched the web without success as browsers seem to use the full url as caching key so I am really curious how I can, under the given circumstances (nodejs, knox library) get the issue solved and use caching control while still being able to generated signed urls for each and every request as I need to verify the user's access rights.

I cannot believe there's no solution to that though.

Answer 1

I am working with Java AmazonS3 client, but the process should be the same.

There is a strategy that can be used to handle this situation.

You could use a fixed date time as an expiration date. I set this date to tomorrow at 12 pm.

Now every time you generate a url, it will be the same throughout that day until 00:00. That way browser caching can be used to some extent.

Answer 2

If you use CloudFront with S3, you can use a Custom Policy, if you restrict each url to the user's IP and a reasonably long timeout, it means that when they request the same content again, they will get the same URL and hence their browser can cache the content but the URL will not work for someone else (on a different IP).

(see: http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-creating-signed-url-custom-policy.html)

Answer 3

Expanding @semir-delji? Answer.

Every time we call getSignedUrl function, it will generate new URLs. This will result in images not being cached even if Cache Control header is present.

Thus, we are using timekeeper library to freeze time. Now when the function is called, it thinks that the time has not passed, and it returns same URL.

const moment = require('moment');
const tk = require("timekeeper");

function url4download(awsPath, awsKey) {

  function getFrozenDate() {
    return moment().startOf('week').toDate();
  }

  // Paramters for getSignedUrl function
  const params = {
    // Ref: https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html
    // Ref: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html
    Bucket: awsBucket,
    Key: `${awsPath}/${awsKey}`,
    // 604800 == 7 days
    ResponseCacheControl: `public, max-age=604800, immutable`,
    Expires: 604800, // 7 days is max
  };

  const url = tk.withFreeze(getFrozenDate(), () => {
    return S3.getSignedUrl('getObject', params);
  });
  return url;
}

Note: Using moment().toDate(), as the timekeeper requires a Native Date Object.

Even tough the question is for using knox library, my answer uses aws official library.

// This is how the AWS & S3 is initiliased.
const AWS = require('aws-sdk');

const S3 = new AWS.S3({
  accessKeyId: awsAccessId,
  secretAccessKey: awsSecretKey,
  region: 'ap-south-1',
  apiVersion: '2006-03-01',
  signatureVersion: 'v4',
});

Inspiration: https://advancedweb.hu/cacheable-s3-signed-urls/

Answer 4

When calculating signed URL you can set 'signingDate' to a fixed moment in the past e.g. yesterday morning, then calculate expiration from that moment. Don't forget to use UTC and account for timezones.

import { S3Client, GetObjectCommand, GetObjectCommandInput } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";

let signingDate = new Date();
signingDate.setUTCHours(0, 0, 0, 0);
signingDate.setUTCDate(signingDate.getUTCDate() - 1);

let params: GetObjectCommandInput = {
    Bucket: BUCKET_NAME,
    Key: filename
};
const command = new GetObjectCommand(params);
const url = await getSignedUrl(s3Client,
    command, {
        expiresIn: 3 * 3600 * 24, // 1 day until today + 1 expiration + 1 days for timezones
        signableHeaders: new Set < string > (),
        signingDate: signingDate
    });