AzP: Terraform modules private repo

So Terraform is all the rage at the moment, so why not cash in on some of that action?

So here is the pitch, write IaC with Terraform you should at least go down the path of creating modules and reference them in your code. This is a good move and something we know from previous experience with other languages makes everything more maintainable as entropy tries to run us over.

Though mixing modules and running terraform in a pipeline some new challenges arise. Especially if you want to host the modules in a private repository.

The Microsoft backed tasks for running Terraform in a AzD pipeline needs in it’s init phase to resolve and download all external referenced modules. This is not a problem for the ones that are public, but if you have module in a private repo then this phase will just fail.

So a solution is to host the modules in AzD git and leverage the access token the agent is given when running a pipeline.

Note:
Depending on the project structure you might need to give the build account you are running the Terraform pipeline under access to read the repo(s) where you host the terraform modules.

Now all you need is to create a powershell task that runs before Terraform init task and pass in the build access token to the powershell script below. This will use the token to configure git on the agent to use it to access repos where the modules are. If you have your own custom agents, make sure git is installed on all of them for this to work correctly.

Now for the code to set the git context:

<#
.SYNOPSIS
Uses AzD agent Oauth token to set git config extraheader so terraform init can use private AzD git repo as module source
.DESCRIPTION
Section running this task needs to have access to Oauth token. This is configured on the agent step in the pipeline config.
.PARAMETER AzDteamAccountURL
Azure DevOps account url
https://AccountName@dev.azure.com/AccountName
.PARAMETER AccessToken
Access token with access rights to run git clone on private repos that contain modules terraform code uses
.PARAMETER Verbose
If set will print git system config content to log
.NOTES
AUTHOR: Morten Lerudjordet
#>
[CmdletBinding()]
param(
[Parameter(Mandatory = $true)]
[ValidateNotNullorEmpty()]
[string]$AzDaccountURL,
[Parameter(Mandatory = $true)]
[ValidateNotNullorEmpty()]
[string]$AccessToken
)
try
{
Write-Host Object "Task started at time: $(get-Date format r).`nRunning PS version: $($PSVersionTable.PSVersion)`nOn agent: $($env:computername)"
#region Checks
# Check if agent is allowed to use access token
if($AccessToken -eq '$(System.AccessToken)')
{
Write-Error Message "Agent has not been allowed to use Oauth Token (System.AccessToken)" ErrorAction Stop
}
# Check if git is installed on agent
if(-not (Test-Path Path "$env:ProgramFiles\Git\cmd\git.exe"))
{
Write-Error Message "Git is missing from default location: $($env:ProgramFiles)\Git\cmd\git.exe on agent pipeline is running" ErrorAction Stop
}
#endregion
#region Variables
# Put temp git config in agent temp dir so it will be cleaned up
$GitTempConfigFileName = Join-Path Path $($env:AGENT_TEMPDIRECTORY) ChildPath ".gitconfig.$($env:Release_ReleaseId)"
#endregion
Write-Host Object "Setting git extraheader config file for $AzDaccountURL with AzD access token"
# Workaround to get powershell to run git command, git must be installed in default location on agents
$GitCommand = "& `"$env:ProgramFiles\Git\cmd\git.exe`" config –file $GitTempConfigFileName http.$AzDaccountURL.extraheader `"AUTHORIZATION: bearer $AccessToken`""
$GitCommandResult = Invoke-Expression Command $GitCommand ErrorAction SilentlyContinue ErrorVariable oErr
if($oErr)
{
Write-Error Message "Failed to run git command to set authorization header for: $AzDaccountURL.`nError: $($oErr.Message)" ErrorAction Stop
}
if($GitCommandResult)
{
Write-Host Object "Output from git command:"
Write-Host Object $GitCommandResult
Write-Error Message "Unexpected output from setting extraheader for $AzDaccountURL using AzD access token" ErrorAction Stop
}
else
{
Write-Host Object "Successfully set extraheader in git config file"
}
$GitCommand = "& `"$env:ProgramFiles\Git\cmd\git.exe`" config –global include.path $GitTempConfigFileName"
$GitCommandResult = Invoke-Expression Command $GitCommand ErrorAction SilentlyContinue ErrorVariable oErr
if($oErr)
{
Write-Error Message "Failed to run git command for including temp extraheader config.`nError: $($oErr.Message)" ErrorAction Stop
}
if($GitCommandResult)
{
Write-Host Object "Output from git command:"
Write-Host Object $GitCommandResult
Write-Error Message "Unexpected output from setting temp global extraheader config for: $AzDaccountURL" ErrorAction Stop
}
else
{
Write-Host Object "Successfully added config file to global config"
}
if($VerbosePreference -ne 'SilentlyContinue')
{
Write-Host Object "Getting list of git global config from agent"
$GitCommand = "& `"$env:ProgramFiles\Git\cmd\git.exe`" config –global –list"
$GitCommandResult = Invoke-Expression Command $GitCommand ErrorAction SilentlyContinue ErrorVariable oErr
if($oErr)
{
Write-Error Message "Failed to run git command to list global config: $AzDaccountURL.`nError: $($oErr.Message)" ErrorAction Stop
}
if($GitCommandResult)
{
Write-Host Object "List from global config:"
Write-Host Object $GitCommandResult
}
}
}
catch
{
if ($_.Exception.Message)
{
Write-Error Message "$($_.Exception.Message)" ErrorAction Continue
Write-Host Object "##[error]$($_.Exception.Message)"
}
else
{
Write-Error Message "$($_.Exception)" ErrorAction Continue
Write-Host Object "##[error]$($_.Exception)"
}
}
finally
{
Write-Host Object "Task ended at time: $(get-Date format r)"
}
view raw Set-GitContext.ps1 hosted with ❤ by GitHub

So with this done you will want to clean up the context after the terraform init task has run with the following code:

<#
.SYNOPSIS
Removes git extraheader for AzD project from pipeline after terraform has feteched modules from AzD repo
.DESCRIPTION
.PARAMETER AzDteamAccountURL
Azure DevOps account url
https://dev.azure.com/AccountName
.PARAMETER Verbose
If set will print git system config content to log
.NOTES
AUTHOR: Morten Lerudjordet
#>
[CmdletBinding()]
param(
[Parameter(Mandatory = $true)]
[ValidateNotNullorEmpty()]
[string]$AzDaccountURL
)
try {
Write-Host Object "Task started at time: $(get-Date format r).`nRunning PS version: $($PSVersionTable.PSVersion)`nOn agent: $($env:computername)"
$GitTempConfigFileName = ".gitconfig.$($env:Release_ReleaseId)"
if ($VerbosePreference -ne 'SilentlyContinue') {
Write-Host Object "Getting list of git global config from agent"
$GitCommand = "& `"$env:ProgramFiles\Git\cmd\git.exe`" config –global –list"
$GitCommandResult = Invoke-Expression Command $GitCommand ErrorAction SilentlyContinue ErrorVariable oErr
if ($oErr) {
Write-Error Message "Failed to run git command to list global config: $AzDaccountURL.`nError: $($oErr.Message)" ErrorAction Stop
}
if ($GitCommandResult) {
Write-Host Object "List from global config:"
Write-Host Object $GitCommandResult
}
}
Write-Host Object "Removing git extraheader from agent config for $AzDaccountURL"
$GitCommand = "& `"$env:ProgramFiles\Git\cmd\git.exe`" config –global –unset include.path `"$GitTempConfigFileName`""
$GitCommandResult = Invoke-Expression Command $GitCommand ErrorAction SilentlyContinue ErrorVariable oErr
if ($oErr) {
Write-Error Message "Failed to run git command for clearing extraheader.`nError: $($oErr.Message)" ErrorAction Stop
}
if ($GitCommandResult) {
Write-Host Object "Output from git command:"
Write-Host Object $GitCommandResult
Write-Error Message "Unexpected output from clearing extraheader for $AzDaccountURL" ErrorAction Stop
}
else {
Write-Host Object "Successfully removed temp git config file from global config"
}
if ($VerbosePreference -ne 'SilentlyContinue') {
Write-Host Object "Getting list of git global config from agent"
$GitCommand = "& `"$env:ProgramFiles\Git\cmd\git.exe`" config –global –list"
$GitCommandResult = Invoke-Expression Command $GitCommand ErrorAction SilentlyContinue ErrorVariable oErr
if ($oErr) {
Write-Error Message "Failed to run git command to list global config: $AzDaccountURL.`nError: $($oErr.Message)" ErrorAction Stop
}
if ($GitCommandResult) {
Write-Host Object "List from global config:"
Write-Host Object $GitCommandResult
}
}
}
catch {
if ($_.Exception.Message) {
Write-Error Message "$($_.Exception.Message)" ErrorAction Continue
Write-Host Object "##[error]$($_.Exception.Message)"
}
else {
Write-Error Message "$($_.Exception)" ErrorAction Continue
Write-Host Object "##[error]$($_.Exception)"
}
}
finally {
Write-Host Object "Task ended at time: $(get-Date format r)"
}

Hope this helps anybody else struggling with this.

Happy tinkering!

Leave a Comment

Your email address will not be published. Required fields are marked *