EKS Terraform - datasource aws_eks_cluster_auth - token expired

Question

I've an EKS cluster deployed in AWS and I use terraform to deploy components to that cluster.

In order to get authenticated I'm using the following EKS datasources that provides the cluster API Authentication:

data "aws_eks_cluster_auth" "cluster" {
  name = var.cluster_id
}

data "aws_vpc" "eks_vpc" {
  id = var.vpc_id
}

And using the token inside several local-exec provisioners (apart of other resources) to deploy components

resource "null_resource" "deployment" {
  provisioner "local-exec" {
    working_dir = path.module
    command     = <<EOH
kubectl \
  --server="${data.aws_eks_cluster.cluster.endpoint}" \
  --certificate-authority=./ca.crt \
  --token="${data.aws_eks_cluster_auth.cluster.token}" \
  apply -f test.yaml
EOH
  }
}

The problem I have is that some resources are taking a little while to deploy and at some point when terraform executes the next resource I get this error because the token has expired:

exit status 1. Output: error: You must be logged in to the server (the server has asked for the client to provide credentials)

Is there a way to force re-creation of the data before running the local-execs?

Answer 1

The data.aws_eks_cluster_auth.cluster_auth.token creates a token with a non-configurable 15 minute timeout.

One way to get around this is to use the sts token to create a long-lived service-account token and use that to provision the terraform-kubernetes-provider for long running kuberenetes resources.

I created a module called terraform-kubernetes-service-account to capture this common behavior of creating a service account, giving it some permissions, and output the auth information i.e. token, ca.crt, namespace.

For example:

module "terraform_admin" {
  source = "aidanmelen/service-account/kubernetes"

  name               = "terraform-admin"
  namespace          = "kube-system"
  cluster_role_name  = "terraform-admin"
  cluster_role_rules = [
    {
      api_groups     = ["*"]
      resources      = ["*"]
      resource_names = ["*"]
      verbs          = ["*"]
    },
  ]
}

provider "kubernetes" {
  alias                  = "terraform_admin_service_account"
  host                   = "https://kubernetes.docker.internal:6443"
  cluster_ca_certificate = module.terraform_admin.auth["ca.crt"]
  token                  = module.terraform_admin.auth["token"]
}

data "kubernetes_namespace_v1" "example" {
  metadata {
    name = kubernetes_namespace.ex_complete.metadata[0].name
  }
}