Django migrate won't work due to Postgres InsufficientPrivilege error provisioned by terraform and helm

Question

We're are hitting a issue with our Django app and are unable to find the underlying problem.

Our Django app runs on Kubernetes and is managed by Helm. When we upgrade the app, a Helm upgrade job is triggered that makes sure that a manage.py migrate is run. The migrate job runs with database admin privileges, not the customers postgres role.

The error we're getting is: InsufficientPrivilege: permission denied for table RouteInstance

This error must have something to do with creating a reference from a new table to a existing table, but we can't find it. Maybe it's in the Terraform resources config, or are the grants not sufficient. We don't use any extra grants since the admin account should be a real admin (Digitalocean doadmin).

Any help would be awesome, we're stuck at the moment.

Some context on our app deployment:

The Helm template is being deployed by Terraform, the application deployment consists of:

• Create a subdomain record
• Create a namespace
• Create S3 buckets and a account
• Deploy the Helm chart (helm_release), here we use a values list with a templatefile. Between these values there are variables:
  • postgres_user_username (used to run the app)
  • postgres_user_password
  • postgres_admin_username (used to perform the helm upgrade job containing the migrate)
  • postgres_admin_password
• Create a postgresql_role for the customer (their app runs under the customers postgres role)
• Create a postgresql_database for the app
• Set postgresql_default_privileges (table, sequence), see config below

Terraform resouce configs:

resource "postgresql_role" "postgres_role" {
  name = "customer_user"
  login = true
  password = "redacted..."
}

resource "postgresql_database" "app_database" {
  name = "app_database"
  owner = postgresql_role.postgres_role.name
  depends_on = [ postgresql_role.postgres_role ]
}

resource "postgresql_default_privileges" "postgres_privileges_database" {
  role = postgresql_role.postgres_role.name
  database = postgresql_database.app_database.name
  schema = "public"

  owner = postgresql_role.postgres_role.name
  object_type = "table"
  privileges = ["SELECT", "INSERT", "UPDATE", "DELETE"]
}

resource "postgresql_default_privileges" "postgres_privileges_sequence" {
  role = postgresql_role.postgres_role.name
  database = postgresql_database.app_database.name
  schema = "public"

  owner = postgresql_role.postgres_role.name
  object_type = "sequence"
  privileges  = ["USAGE", "SELECT"]
}

  resource "postgresql_grant" "postgres_grant_database" {
  database = postgresql_database.app_database.name
  role = postgresql_role.postgres_role.name
  object_type = "database"
  privileges = ["CONNECT"]
}

resource "postgresql_grant" "postgres_grant_tables" {
  database = postgresql_database.app_database.name
  role = postgresql_role.postgres_role.name
  schema = "public"
  object_type = "table"
  privileges = ["SELECT", "INSERT", "UPDATE", "DELETE"]
}

resource "postgresql_grant" "postgres_grant_sequences" {
  database    = postgresql_database.app_database.name
  role        = postgresql_role.postgres_role.name
  schema      = "public"
  object_type = "sequence"
  privileges  = ["USAGE", "SELECT", "UPDATE"]
}