Salesforce, Python, SQL, & other ways to put your data where you need it

Need event music? 🎸

Live and recorded jazz, pop, and meditative music for your virtual conference / Zoom wedding / yoga class / private party with quality sound and a smooth technical experience

Variables and secrets for CI/CD pipelines

03 Aug 2023 🔖 devops integration git azure
💬 EN

Table of Contents

Azure DevOps Pipelines and GitHub Actions both let you store variables and secrets in their services, for use with your CI/CD automations, but they’ve arranged them in pretty different parts of their settings options.

For the purposes of this article, I’m declaring the following definitions:

  1. A “CI/CD variable” is a pair of pieces of plaintext, the first being used as a “key” and the second as a “value,” where neither key nor value are important to hide from people with access to edit the value.

  2. A “CI/CD secret” is a pair of pieces of plaintext, again with the first as a “key” and the second as a “value,” where the contents of the value must not be visible to a human, once saved, by revisiting the user interface for editing the value.

Azure DevOps

Azure DevOps (“ADO”) calls both CI/CD variables and CI/CD secrets “variables,” and simply puts a checkbox into the user interface for editing them asking if you want the value to be kept a secret.

There are two places to store CI/CD variables/secrets in ADO:

  1. Pipeline variables
    • (Relatively comparable to GitHub Actions per-repository variables/secrets. While ADO pipelines exist as top-level sub-items of an ADO “project,” not as sub-items of an ADO repository within an ADO project, any given ADO pipeline is typically only meant to run as a CI/CD automation for changes to the contents of one ADO repository at a time.)
  2. Variable Group variables
    • (Helps add name-scope granularity to variables that are shareable amongst multiple pipelines in an ADO “project,” much like GitHub “environments” add to GitHub repositories.)
      • (That said, GitHub “environments” are also used to help restrict CI/CD processes – for example, requiring manual approval to deploy to production. In ADO, these two concepts are separated. ADO variable-name-scoping should be configured “Pipelines -> Libraries -> Variable Groups” and is documented below. ADO CI/CD process controls should be configured under a completely different part of the user interface at “Pipelines -> Environments” for ADO YAML deployment pipelines and at “Pipelines -> Releases” for ADO classic deployment pipelines.)

You can also store your CI/CD variables and CI/CD secrets outside of ADO:

  1. In Azure App Configuration
  2. In Azure Key Vault

Finally, you can avoid having to worry about so many secrets with:

  1. Azure Managed Identity

Pipeline variables

Writing key-value pairs

CI/CD variables

Within a specific pipeline within an ADO project, you might want to create CI/CD variables whose “key” name is scoped specifically to that pipeline.

For example, “day_week_starts_on.”

YAML

When coding a YAML pipeline, you can either define the CI/CD variable as part of your pipeline code (which will make it version-controlled with Git) or store it outside of your codebase within your ADO pipeline’s settings in the editor’s “Variables” button (which will not be Git-tracked).

Classic

When building a CI/CD pipeline in ADO’s point-and-click “classic” editor, you can define the CI/CD variable in the editor’s “Variables” tab.

CI/CD secrets
YAML

Caution: When coding a YAML pipeline, do not define the CI/CD secret as part of your pipeline code! Your pipeline code is tracked with Git version control, and you don’t ever want to store the value of sensitive secrets inside code – particularly version-tracked code whose history is hard to clean up.

For YAML pipelines, store CI/CD secrets of your codebase within your ADO pipeline’s settings in the editor’s “Variables” button, being sure to toggle them to be secret.

Classic

For classic pipelines and “release” pipelines, store CI/CD secrets in the editor’s “Variables” tab, being sure to toggle them to be secret.

Reading key-value pairs

YAML

When coding a CI/CD pipeline in ADO’s YAML syntax, you can reference the pipeline’s variables with a syntax such as “$(day_week_starts_on).” (Or, in a task with its own scripting environment like a PowerShell task, perhaps with a syntax such as “${env:day_week_starts_on}.”)

You might have to do one more step first if the pipeline variable is secret.

Classic

When building a CI/CD pipeline in ADO’s point-and-click “classic” editor, you should be able to easily reference the pipeline’s variables in the pipeline editor.

You might have to do one more step first if the pipeline variable is secret.

Variable group variables

Official Microsoft documentation: “Add & use variable groups”

Writing key-value pairs

CI/CD variables

Within any given ADO project, you can create a number of “variable groups.”

  • For example, you might create ones called “animalfacts_prod,” “animalfacts_staging,” “animalfacts_dev,” etc.
    • Then, repeatedly within each of these, you might create three variables called “azure_app_service_resource_name,” “azure_app_service_resource_group_name,” and “enable_monitoring.”
  • You might also create one called “projectwide” and give it a few variables with names like “corporate_primary_domain_name.” Or you might create one called “animalfacts_all” and give it a few variables with names like “day_week_starts_on.”
    • (Note: If you’re only going to reference “day_week_starts_on” from one pipeline, and you’re never going to need to change its value depending on context like “prod” or “staging” or “dev,” you might prefer storing it as a pipeline variable.)

Best practice: Because ADO variable groups (and, consequently, the variables within) exist at the ADO project level, not the ADO repository level, descriptive variable group names (and filling out the “description” area) are strongly recommended.

CI/CD secrets

You can make such a variable “write-only” within this editing user interface by clicking the padlock icon to the right of the variable’s key and value. (This icon’s hover-tooltip reads “change variable type to secret.”)

Key Vault

You can also make your variable group serve as a handy cross-reference to an Azure Key Vault resource, so that your CI/CD pipelines can be less verbose when accessing Key Vault.

Simply toggle on “Link secrets from an Azure key vault as variables” when editing the variable group you’ve created.

Caution: Any given variable group can’t simultaneously be a cross-reference to Key Vault and have variables of its own at the same time. Any variables you have already created in the variable group will be wiped out when you turn on the toggle.

Reading key-value pairs

Official Microsoft documentation: “Use a variable group’s secret and nonsecret variables in an Azure Pipeline”

YAML

When coding a CI/CD pipeline in ADO’s YAML syntax, you can reference these variables with a syntax such as “$(azure_app_service_resource_name)” or “$(day_week_starts_on)” by specifying the name of the variable group in which they live.

Caution: If you were to specify both “animalfacts_prod” and “animalfacts_staging,” in that order, in the same level of variable scope inside the YAML, “$(azure_app_service_resource_name)” would refer to the value of the “staging” version because you specified it last.

Protection

By default, no “YAML” pipelines in an ADO project can access the values of variable groups in your project, even if they reference them in the YAML code.

You must explicitly add each “YAML” pipeline that will need to reference the variable group in the variable group’s “Pipeline permissions” tab before YAML pipelines referencing variables from the variable group will execute correctly.

Classic

When building a CI/CD pipeline in ADO’s point-and-click “classic” editor, you can import additional variables into the pipeline’s scope under the “Variables” tab of the pipeline editor (the same place that you would add variables to the “classic” pipeline itself) by “linking” variable groups to the pipeline.

Protection

Any “classic” pipeline or “classic” release-pipeline in an ADO project is allowed to be linked to any variable group in the project. You cannot explicitly restrict variable group access to a specific set of “classic” pipelines, so be careful who you let create and edit “classic” pipelines in your ADO project.

(If they create a “classic” pipeline that dumps out the value of a secret key-value pair into a human-readable log, they can exfiltrate your secret.)

App Configuration

Rather than storing your CI/CD variables in ADO Pipelines or ADO Variable Groups, you can store them in Azure App Configuration.

Here’s how you read App Configuration values back out into an ADO pipeline you’re building.

(You can also write values into App Configuration from an ADO pipeline’s runtime!)

Note: For CI/CD secrets, the value of the Azure App Configuration’s key is typically a pointer to some key name in an Azure Key Vault resource, the value of which subsequently has to be fetched the way you would fetch a Key Vault resource directly by name.

That extra layer of abstraction may not be terribly useful for ADO Pipelines, but it exists because Microsoft’s SDKs for writing long-running web applications are able to more easily watch App Configuration for changes than watch Key Vault.

That said, the extra layer of abstraction does let you separate “write” permissions to secrets.

You could, for example, have a Key Vault key named “secret_for_2020” that only 2 humans have write access to the value of.

And then you could have an Azure App Configuration key named “current_secret” pointing to that Key Vault resource, where the App Configuration key’s value is “secret_for_2020.” Perhaps 10 humans have write access to the App Configuration resource, letting them update it to “secret_for_2021” on January 1, 2021.

(Obviously this is a ridiculous example, but hopefully it gets the point across about how adding a layer of abstraction can help break up security models.)

Key Vault

Although Microsoft now recommends storing non-secret key-value pairs in Azure App Configuration rather than Key Vault, and also recommends adding a layer of abstraction onto Key Vault with App Configuration (see notes in section above), you certainly can store both CI/CD variables and CI/CD secrets in Azure Key Vault alone.

Here’s how to read Key Vault values back out into an ADO pipeline you’re building.

(Don’t forget, though, that you can also set up a Variable Group as a handy short-reference to a KeyVault resource.)

Managed Identity

Don’t forget that you might be able to use Azure Active Directory Managed Identity to help ADO pipelines read from, write to, or deploy to various Azure resources.

This is always preferable to trying to keep secrets safe if you can get it to work.

Ask around at your company if you don’t know how to get it to work.

GitHub

There are three places to store CI/CD variables/secrets in GitHub:

  1. Within settings for an entire organization
    • (No equivalent available in ADO. The least-granular variable/secret scope available in ADO is at the “project” level via “variable groups.”)
  2. Within settings for a given repository
    • (Relatively comparable to ADO per-pipeline variables. While ADO pipelines exist as top-level sub-items of an ADO “project,” not as sub-items of an ADO repository within an ADO project, any given ADO pipeline is typically only meant to run as a CI/CD automation for changes to the contents of one ADO repository at a time.)
  3. Name-scoped as a sub-level of a given repository (GitHub calls it an “environment“).
    • (Note: GitHub “environments” are also used to help restrict CI/CD processes – for example, requiring manual approval to deploy to production. In ADO, these two concepts are separated. ADO variable-name-scoping should be configured “Pipelines -> Libraries -> Variable Groups” and is documented above. ADO CI/CD process controls should be configured under a completely different part of the user interface at “Pipelines -> Environments” for ADO YAML deployment pipelines and at “Pipelines -> Releases” for ADO classic deployment pipelines.)

You can also store your CI/CD variables and CI/CD secrets outside of GitHub:

  1. In Azure App Configuration
  2. In Azure Key Vault

Finally, you can avoid having to worry about so many secrets with:

  1. OpenID Connect (“OIDC”) (see also this second article)

Organization variables and secrets

Writing key-value pairs

Create, edit, and delete key-value pairs in the organization’s settings under “Secrets and Variables -> Actions,” in the “Secrets” tab for a write-only user interface and in the “Variables” tab for non-secret information.

For example, “corporate_primary_domain_name.”

Reading key-value pairs

CI/CD variable example: “${{ vars.CORPORATE_PRIMARY_DOMAIN_NAME }}”

CI/CD secret example: “${{ secrets.THIS_SEEMS_LIKE_A_BAD_IDEA }}”

Repository variables and secrets

You might want to create CI/CD variables whose “key” name is scoped specifically to CI/CD pipelines running against a certain repository.

For example, “day_week_starts_on.”

Writing key-value pairs

Create, edit, and delete key-value pairs in the repository’s settings under “Secrets and Variables -> Actions,” in the “Secrets” tab for a write-only user interface and in the “Variables” tab for non-secret information.

Reading key-value pairs

CI/CD variable example: “${{ vars.DAY_WEEK_STARTS_ON }}”

CI/CD secret example: “${{ secrets.STILL_PROBABLY_TOO_BROADLY_SCOPED }}”

Environment variables and secrets

Caution: Not be confused with the idea of a running CI/CD pipeline’s operating system’s “env” variables, also known as “environment variables.”

Writing key-value pairs

Within any given GitHub repository, you can create a number of “environments.”

  • For example, you might create ones called “prod,” “staging,” “dev,” etc.
    • Then, repeatedly within each of these, you might create three secrets or variables called “azure_app_service_resource_name,” “azure_app_service_resource_group_name,” and “enable_monitoring.”

Reading key-value pairs

Within any block of YAML that has an “environment” property containing a value like “prod,” “staging,” “dev,” etc. (or with a sub-property of “name” containing such a value):

CI/CD variable example: “${{ vars.ENABLE_MONITORING }}”

CI/CD secret example: “${{ secrets.PLEASE_TRY_OIDC_NOT_SECRETS_THANK_YOU }}”

(In all seriousness, you might decide to make your resource name a secret rather than a variable, even though it’s not necessarily so much a secret-secret. So perhaps you’d still read it out as “${{ secrets.AZURE_APP_SERVICE_RESOURCE_NAME }}.“)

App Configuration

Rather than storing your CI/CD variables in ADO Pipelines or ADO Variable Groups, you can store them in Azure App Configuration.

TODO: Figure out how to read App Configuration values back out into a GitHub Action. (It looks like Microsoft doesn’t publish an official App-Configuration-reading GitHub Action to follow after its OIDC-enabled “Azure Login” action the way they publish an official App Configuration task for ADO? Maybe the next-best thing is fetching such a key-value pair with their official Azure PowerShell GitHub Action?)

Note: For CI/CD secrets, the value of the Azure App Configuration’s key is typically a pointer to some key name in an Azure Key Vault resource, the value of which subsequently has to be fetched the way you would fetch a Key Vault resource directly by name.

That extra layer of abstraction may not be terribly useful for GitHub Actions, but it exists because Microsoft’s SDKs for writing long-running web applications are able to more easily watch App Configuration for changes than watch Key Vault.

That said, the extra layer of abstraction does let you separate “write” permissions to secrets.

You could, for example, have a Key Vault key named “secret_for_2020” that only 2 humans have write access to the value of.

And then you could have an Azure App Configuration key named “current_secret” pointing to that Key Vault resource, where the App Configuration key’s value is “secret_for_2020.” Perhaps 10 humans have write access to the App Configuration resource, letting them update it to “secret_for_2021” on January 1, 2021.

(Obviously this is a ridiculous example, but hopefully it gets the point across about how adding a layer of abstraction can help break up security models.)

Reminder: Be sure to use OIDC rather than a secret when communicating with App Configuration or Key Vault, wherever possible.

Key Vault

Although Microsoft now recommends storing non-secret key-value pairs in Azure App Configuration rather than Key Vault, and also recommends adding a layer of abstraction onto Key Vault with App Configuration (see notes in section above), you certainly can store both CI/CD variables and CI/CD secrets in Azure Key Vault alone.

TODO: Figure out how to read Key Vault values back out into a GitHub Action. (It looks like Microsoft doesn’t publish an official App-Configuration-reading GitHub Action to follow after its OIDC-enabled “Azure Login” action the way they publish an official Key Vault task for ADO? Maybe the next-best thing is fetching such a key-value pair with their official Azure PowerShell GitHub Action?)

Reminder: Be sure to use OIDC rather than a secret when communicating with Key Vault, wherever possible.

OIDC

Don’t forget that you might be able to use OIDC to help GitHub Actions read from, write to, or deploy to various Azure resources.

This is always preferable to trying to keep secrets safe if you can get it to work.

Ask around at your company if you don’t know how to get it to work.

Takeaway

Azure DevOps Pipelines and GitHub Actions offer very similar ways of storing and referencing variables and secrets, but it’s not obvious that they do, since the user interfaces are so different between the two services.

However, you should be able to more or less find whatever you need in either service. You just might have to organize and name things a little differently between the two of them.

--- ---