Azure DevOps Pipeline Tips and Tricks

This is the bookend to a series of posts about Azure DevOp YAML pipelines. This will be a living document of Tips and Tricks.

First post in the series: CI/CD YAML Pipelines

Tips

Always manually run a new or updated pipeline off a branch since you will be running it many times to get it to work
When editing in the AzDO UI, use Validate in the kebab menu on the web to catch many types of errors before doing a Run
When editing in the AzDO UI, use Download full YAML in the kebab menu to get the full expanded YAML of a pipeline. This is before a run, so it will use default parameters.
After a pipeline runs, you can use the kebab menu Download logs to download a zip file of all the logs and the azure-pipelines-expanded.yml file which shows the YAML after all templates have been expanded.
To re-run a failed pipeline when some environment change should have fixed it (no code or YAML change), you can Rerun failed jobs to rerun the same code.
To re-run a pipeline with new code and the same parameters as before, click Run New from the pipeline to avoid re-entering any custom parameters.
To get more verbose logging when manually running a pipeline, check the Enable system diagnostics checkbox. You can also add this task to dump a file listing and env vars at any point in a pipeline: - template: /templates/dump-environment.yml@loyal-templates
When adding a shell script to YAML, at least drop the script into an editor like VSCode with syntax highlighting to catch syntax errors. Better yet, run it locally before dropping it into the YAML.

Tricks

Add DryRun Option

Add a dryRun parameter to your pipeline defaulting to false. Then when manually running it, you can set it to true to see what would happen without actually doing it.

For builds, this should build and run unit tests, but not push the artifact.
For K8s deploys, this should dump the K8s manifests to the logs so you can review them to make sure the values are what you expect. In particular, look for $( which indicates that a name inside the parens isn’t set, e.g. $(UnSetName)

Watching for $(…) in Output

When I deploy to K8s use Helm, I usually dump out the manifests in a dry run. You can validate what will be deployed before you deploy it. When looking this output there should be no $(myVar) variables. If there are, that means that the variable myVar is not set in the pipeline and will be sent to K8s as-is, which will cause an error.

You can use this technique for any text that is generated in the pipeline.

Using DevOps Variables in Scripts

If you have a bash or PowerShell script in your pipeline and want to use variables or parameters, you have to be careful since they are replaced verbatim. Both of these will work fine in most cases, but what if MyVar or MyParam contains a $ or "? Then you will get a syntax error at runtime. Passwords can wreak havoc.

- pwsh: |
    Write-Host "MyVar is $(MyVar)"
    Write-Host "MyParam is $"
  name: Echo it dangerously

It’s usually best to pass in values as environment variables like this:

- pwsh: |
    Write-Host "MyVar is $env:MyVar"
    Write-Host "MyParam is $env:MyParam"
  name: Echo it safely
  env:
    MyVar: $(MyVar)
    MyParam: $

Dumping Out Files and Environment Variables

This is the YAML for a step template that dumps the directory and environment variables when a pipeline runs. I’ve found this handy for debugging or just exploring what variables are available in the pipeline. You will never see secrets in the environment variables. To get a secret into a shell script, you can use the env: parameter to pass it in.

parameters:
  - name: 'directory'
    displayName: 'Directory to dump'
    type: string
    default: $(Agent.BuildDirectory)

steps:
  - pwsh: |
      Write-Host "##[group]PS Version"
      $PSVersionTable | out-string -width 1000
      Write-Host "##[endgroup]"

      Write-Host "##[group]Files"
      Get-ChildItem $env:dir -Recurse -Force| Where-Object FullName -notlike '*/.git*' | Select-Object FullName | Sort-Object | Out-String -width 1000
      Write-Host "##[endgroup]"

      Write-Host "##[group]Environment"
      Get-ChildItem env: | Select-Object key,value | Out-String -width 1000 | Sort-Object
      Write-Host "##[endgroup]"
    displayName: Dump Version, Dir, and Env
    env:
      dir: ${{ parameters.directory }}

Manually Running a Pipeline with the Same Settings

If you change the branch or parameters on a manual run of a pipeline, to avoid repeating the same settings, click Run New from the finished pipeline run and it will pre-fill the settings for you.

Adding Multiple YAML Pipelines at Once

If you’re adding multiple pipelines in the AzDO UI from one repo, after adding the first one use the back button in the browser to get back to YAML selection list and save a few clicks.

Troubleshooting

My Parameter Value Isn’t Used

Spelling counts, but misspelling is ignored. For example, if you a task take a parameter myParam and you pass in myParm is will not warn you about myParm. If myParam is required it will bark about that one missing.

My Pipeline Isn’t in the List of Pipelines

The default Pipelines view is the Recent tab. If you’ve never run it yet, it will be in the All tab.

My Build Pipeline Isn’t Getting Triggered

Check your triggers: section in the build.yml It restricts which branches and files will trigger a new build.

Edit the pipeline in the browser, and click on Validate from the kebab menu to see if there are any syntax errors in the YAML, which would prevent the pipeline from running.

My Deploy Pipeline Isn’t Getting Triggered

The deploy should only be triggered by a successful run of the build pipeline. Check the resources.pipeline section of the deploy.yml to make sure it is configured correctly. The source must be the exact name of the build pipeline in Azure DevOps. The resources.pipeline.trigger should only trigger on the main branch.

Edit the pipeline in the browser, and click on Validate from the kebab menu to see if there are any syntax errors in the YAML, which would prevent the pipeline from running.

Helm `already exists` error

If you get an error like this that says the resource already exists, it could be a couple issues.

Error: rendered manifests contain a resource that already exists. Unable to continue with install: CronJob "featuretoggles-effective-date-job" in namespace "default" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-name" must equal "my-cool-job": current value is "my-kewl-job"

The resource already exists in the cluster, but was not created by helm.
The resource was created by helm, but the meta.helm.sh/release-name annotation was changed. This could be because the new helm configuration has a different name. You will see this if you split a values.yaml file that did multiple things into separate files, where each one has a different releaseName. A new cron job build and deploy split out of an API build and deploy will get this.

Gotchas

YAML

‘Nuff said.

Using $variables.varName$ in a Template

For variables that ar known at compile-time, instead of the macro syntax ($(varName)) you can use the template syntax like $variables.varName$ . This is helpful when debugging since the expanded YAML will show the value of the variable. However, you cannot use this in a template since the variables are not passed into one. If you expand YAML with a template that uses this syntax, the values will be an empty string.

The basic rule of thumb is that if you declare or load a variable in YAML, you can use the template syntax for it.

PREVIOUSDynamic CI/CD Pipeline

NEXTAzure DevOps Inline Script Tips