Executing Terraform in Azure pipelines fails because of authentication

Question

I want to provision Azure resources via Terraform as part of an Azure pipeline. My terraform files are contained in the terraform folder. Running cat terraform/*.tf gives:

# Actual values are replaced by "example" for security
terraform {
  backend "azurerm" {
    subscription_id = "example"
    resource_group_name = "example"
    storage_account_name = "example"
    container_name = "example"
    key = "example"
  }
  required_providers {
    azurerm = {
      source = "example"
    }
  }
}

provider "azurerm" {
  features {}
}
resource "azurerm_container_registry" "registry" {
  location            = "example"
  name                = "example"
  resource_group_name = "example"
  sku = "example"
  admin_enabled = "example"
  tags = {
    createdBy = "example"
    neededUntil = "example"
    project = "example"
  }
}
resource "azurerm_key_vault" "vault" {
  sku_name = "example"
  resource_group_name = "example"
  tenant_id = "example"
  name = "example"
  location = "example"
}

Running terraform init and terraform apply works when run on my local machine where I am logged in to Azure via az login. Resources are provisioned as expected.

Now I want to run Terraform commands as part of my Azure Pipeline. This is the relevant part of my pipeline:

resources:
- repo: self

variables:
  vmImageName: 'ubuntu-latest'

stages:
- stage: Provision 
  displayName: Provision Resources with Terraform 
  pool: 
    vmImage: $(vmImageName)
  jobs:
  - job: Provision
    displayName: Provision Resources with Terraform 
    steps:
    - task: TerraformCLI@0  # Only contains init task for brevity
      inputs:
        command: 'init'
        workingDirectory: '$(System.DefaultWorkingDirectory)/terraform'
        backendType: 'azurerm'

If I run the pipeline I get the following error:

Starting: TerraformCLI
==============================================================================
Task         : Terraform CLI
Description  : Execute terraform cli commands
Version      : 0.7.8
Author       : Charles Zipp
Help         : 
==============================================================================
/usr/local/bin/terraform version
Terraform v1.1.9
on linux_amd64
+ provider registry.terraform.io/hashicorp/azurerm v3.5.0
##[error]Terraform backend initialization for AzureRM only support service principal authorization
##[error]Terraform backend initialization for AzureRM only support service principal authorization
Finishing: TerraformCLI

If I change my pipeline like this

...
    - task: TerraformTaskV2@2
      inputs:
        provider: 'azurerm'
        command: 'init'
        workingDirectory: '$(System.DefaultWorkingDirectory)/terraform'
        backendServiceArm: 'ARM' # The service principal I created
        backendAzureRmResourceGroupName: 'example'
        backendAzureRmStorageAccountName: 'example'
        backendAzureRmContainerName: 'example'
        backendAzureRmKey: 'example'

the following error pops up:

Starting: TerraformTaskV2
==============================================================================
Task         : Terraform
Description  : Execute terraform commands to manage resources on AzureRM, Amazon Web Services(AWS) and Google Cloud Platform(GCP)
Version      : 2.203.0
Author       : Microsoft Corporation
Help         : [Learn more about this task](https://aka.ms/AAf0uqr)
==============================================================================
/usr/local/bin/terraform init -backend-config=storage_account_name=tfstatemw -backend-config=container_name=containermw -backend-config=key=***ervice.tfstate -backend-config=resource_group_name=rg-bootcamp-moritz-wolff -backend-config=subscription_id=3f56b8b1-6232-47b8-94de-7285a95e0c7f -backend-config=tenant_id=f1640c14-f2cd-4607-b90a-ec03d1b46437 -backend-config=client_id=*** -backend-config=client_secret=***

Initializing the backend...
╷
│ Error: Backend configuration changed
│ 
│ A change in the backend configuration has been detected, which may require
│ migrating existing state.
│ 
│ If you wish to attempt automatic migration of the state, use "terraform
│ init -migrate-state".
│ If you wish to store the current configuration with no changes to the
│ state, use "terraform init -reconfigure".
╵

##[error]Error: The process '/usr/local/bin/terraform' failed with exit code 1
Finishing: TerraformTaskV2

How can I execute my Terraform tasks in the Azure Pipeline?

Answer 1

I believe the issue here is the lack of a Service Principal configured in ADO to talk to Azure. When running locally Terraform is running as your local user. In ADO, this task, is designed to use a Service Principal. Here's a walkthrough on configuring an ADO service Principal.

The documentation for this task also calls out as much:

When executing commands that interact with Azure such as plan, apply, and destroy, the task will utilize an Azure Service Connection to authorize operations against the target subscription. This is specified via the environmentServiceName input

If you are curious on an end to end walkthrough consider this blog post.