Why Go cqlsh can't get a token while keeps running requests good?

Question

My service runs in AWS cluster and uses Keyspaces. The service has a role and gets, AWS_SECRET_ACCESS_KEY, AWS_ACCESS_KEY_ID, AWS_SESSION_TOKEN, AWS_SECURITY_TOKEN env vars.

To connect Keyspaces I do the following:

import (
    "context"
    "time"
    
    credentialsV1 "github.com/aws/aws-sdk-go/aws/credentials"
    ec2rolecredsV1 "github.com/aws/aws-sdk-go/aws/credentials/ec2rolecreds"
    ec2metadataV1 "github.com/aws/aws-sdk-go/aws/ec2metadata"
    aws_sess "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sigv4-auth-cassandra-gocql-driver-plugin/sigv4"
    "github.com/gocql/gocql"
)

type Repository struct {
    session *gocql.Session
    ...
}

func (r *Repository) ConnectCassandra(ctx context.Context) {
    awsSession, err := aws_sess.NewSession()

    creds := credentialsV1.NewChainCredentials(
        []credentialsV1.Provider{
            &credentialsV1.EnvProvider{},
            &ec2rolecredsV1.EC2RoleProvider{
                Client: ec2metadataV1.New(awsSession),
            },
        },
    )

    value, err := creds.Get()

    auth := sigv4.NewAwsAuthenticator()
    auth.SessionToken = value.SessionToken
    auth.AccessKeyId = value.AccessKeyID
    auth.SecretAccessKey = value.SecretAccessKey

    cluster := gocql.NewCluster(Host)

    cluster.Timeout = 10 * time.Second
    cluster.ConnectTimeout = 10 * time.Second
    cluster.Consistency = gocql.LocalQuorum
    cluster.Keyspace = Keyspace
    cluster.Port = Port

    cluster.Authenticator = auth

    r.session, err = cluster.CreateSession()

}

func (r *Repository) GetData() (*Data, error) {
    iter := r.session.Query(...)

    scanner := iter.Scanner()

    for scanner.Next() {
        ...
    }
}

The service periodically requests some data. After some time (~24h) I start getting errors:

2022/02/10 20:29:50 error: failed to connect to XX.XX.XX.XX:9142 due to error: Authentication failure: Session token expired at Wed Feb 09 10:11:42 UTC 2022

Meanwhile requests are running with no errors. Obviously that means that finally gocql driver retrieves a token. But why I get these errors and what's done wrong? What should I change to stop getting and to be sure that everything works as it should?

Answer 1

A SessionToken is a temporary credential that is retrieved using your AccessKeyId and SecretAccessKey. Session Tokens have an expiration, and are only valid for a period of time.

When it expires, any AWS API call made using that token will return that expiration error you are getting. There are a couple of ways to handle this, but effectively you want to 'refresh' the creds you are using by retrieving a new token, and retrying failed calls.

Looking at your code, you may want to define a cred refresh method on your Repository struct that can be called to get new creds again when that specific error is hit, and retry the failed call.

Can read more here: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html