Defining Environment Variables in CodeBuild Buildspec Files

  • by Emre Yilmaz
  • Jan 7, 2021
  • AWSDevOps
  • Istanbul
Defining Environment Variables in CodeBuild Buildspec Files

AWS CodeBuild is the serverless build service of AWS, and until now, I covered various ways of using it in my previous posts. Today, I would like to talk about how to define and use environment variables in your build specification files or, in other words, buildspecs. In a build project, you can assign plain texts to your environment variables, read parameters from AWS Systems Manager Parameter Store, or retrieve secrets from AWS Secrets Manager. I will give examples to each.

Besides, you can also use CodeBuild with CodePipeline as a part of a CI/CD pipeline. We discuss how to do this in my AWS CodePipeline Step by Step course. Therefore, I will also give an example of exporting an environment variable from a build to be able to use it in a later pipeline action.

Why use environment variables in a build project?

AWS CodeBuild works like a terminal in your pipeline. You provide your code as an input artifact, for example, as a source artifact from a CodeCommit repository, and run some commands on it. These commands may be part of a build process or a test action. Therefore, let’s list why you may need to use environment variables in your builds briefly.

  • The first apparent reason is simplifying your build commands and managing your buildspec file efficiently. You can define environment variables as key-value pairs and refer to them in your commands.
  • You may need to load some configuration parameters from AWS Systems Manager Parameter Store before your build commands run.
  • Or, you can even load sensitive parameter values like usernames and passwords from AWS Secrets Manager.

AWS CodeBuild buildspecs support all these options. So next, let me show you examples of them.

Defining a literal environment variable

In a buildspec file, you define environment variables under the env section. You place your simple environment variables by defining the variables section in env. In the example below, we define an S3_BUCKET variable and assign “my-website-bucket” as its value. Then, we reference this value in the build phase, just like a regular environment variable using the dollar ($) sign.

version: 0.2
env:
  variables:
    S3_BUCKET: "my-website-bucket"
phases:
  ...
  build:
    commands:
      - echo "S3 bucket is $S3_BUCKET"
  ...
artifacts:
    ...

The crucial thing you should note here is that you can only assign literal values to the environment variables declared this way. You cannot assign dynamic values at runtime. If you would like to change the value of the S3_BUCKET variable above, you have to change your buildspec file and push your changes to your repository again. So it is like hardcoding parameter values. But it is better than typing the S3 bucket name in all commands needed in the phases section.

Getting environment variable values from AWS Systems Manager Parameter Store

AWS Systems Manager Parameter Store allows you to store your configuration parameters or secrets securely and scalably. It is an AWS-managed, centralized parameter storage. You can both store encrypted or unencrypted parameters in it. You can even define your parameters under a hierarchy such as an organization. Hence, the Parameter Store has many useful features.

For CodeBuild usage, you can define your parameters on Parameter Store, assign them to an environment variable one by one in your buildspec. Then your build containers can load them before executing your commands. To do this, you use the parameter-store section under env and provide key-value pairs where the key would be the local environment variable used by your commands. The value would be the parameter key name on AWS Systems Manager Parameter Store.

For example, in the code below, we read values for a simple my_s3_bucket parameter, a /my_org/s3_bucket parameter which is actually an s3_bucket parameter under /my_org label, and an encrypted db_username parameter from the Systems Manager Parameter Store.

version: 0.2
env:
  parameter-store:
    S3_BUCKET: "my_s3_bucket"
    ORG_BUCKET: "/my_org/s3_bucket"
    DB_USERNAME: "db_username"
  ...

Also, the sample parameters retrieved here would be like below on AWS Systems Manager Parameter Store Console.

Sample parameters on AWS Systems Manager Parameter Store

The IAM service role assigned to your CodeBuild project must have ssm:GetParameters permissions to access these parameters. Their values will be assigned to S3_BUCKET, ORG_BUCKET, and DB_USERNAME environment variables. You can refer them using a dollar sign ($) like $ORG_BUCKET, as you would do while accessing a regular environment variable in your commands.

Reading values from AWS Secrets Manager

AWS Secrets Manager is a central, encrypted secrets manager provided by AWS. Although you can use the Systems Manager Parameter Store to store encrypted values, Secrets Manager offers more options such as seamless integration with Amazon RDS, DocumentDB, Redshift databases, automatic secrets rotation, etc. But it is not free, unlike the Parameter Store.

You define secrets as key-value pairs in JSON format in AWS Secrets Manager. You can store multiple key-value pairs in a single secret. So the access method from a CodeBuild buildspec follows the same pattern.

This time, you define your environment variables under the secrets-manager key inside the env section. Again your key will be the local environment variable which the secret value will be loaded into, and its value will be in secret-name:JSON-key format.

For example, in the buildspec below, we access the username and password keys defined in the my_secret secret on AWS Secrets Manager.

version: 0.2
env:
  secrets-manager:
    DB_PASSWORD: "my_secret:password"
    DB_USER: "my_secret:username"
  ...

The sample secret definition on AWS Secrets Manager Console:

Sample secret on AWS Secrets Manager

The value of the sample secret in the form of key-value pairs:

Sample secret key-value pairs on AWS Secrets Manager

The value of the sample secret in JSON format on AWS Secrets Manager Console:

Sample secret value in JSON on AWS Secrets Manager

As our previous methods, you can reference these values using $DB_PASSWORD and $DB_USER environment variables. But the IAM service role attached to your CodeBuild project must have secretsmanager:GetSecretValue permissions for your secret, which is my_secret in our example.

Environment variables provided by CodeBuild

AWS CodeBuild provides some environment variables specific to the build that is in progress, such as CODEBUILD_BUILD_ID and CODEBUILD_BUILD_NUMBER during the build execution. You can use them for logging purposes or export them if you have other needs in your pipelines. Please find the variables provided by CodeBuild on AWS CodeBuild - Environment Variables Reference.

Exporting variables from a build project

You can export your literal environment variables in your buildspec files to make them reusable in later CodePipeline action configurations but not the ones read from AWS Systems Manager Parameter Store or AWS Secrets Manager. Also, you cannot export environment variables starting with AWS_.

To do this, you need to provide their variable names as a list under env/exported-variables. For example, the buildspec file below exports the custom S3_BUCKET and CodeBuild-specific CODEBUILD_BUILD_ID environment variables.

version: 0.2
env:
  variables:
    S3_BUCKET: "my-website-bucket"
  exported-variables:
    - S3_BUCKET
    - CODEBUILD_BUILD_ID
  ...
artifacts:
    ...

After the CodeBuild project runs, your CodePipeline action history will show them in action details like below.

Build action output on AWS CodePipeline

You will need to assign an output variable namespace in your build action to reference these variables in later CodePipeline actions in the form of #{VariableNameSpace.VariableName}. If you finished my AWS CodePipeline Step by Step course, you would remember our lecture about using action variables on CodePipeline with a CloudFormation staging stack output and manual approval action configuration example. Although the example in this post is different, the usage is similar.

Other options to define environment variables and precedence

In this post, I concentrated on defining environment variables in CodeBuild buildspec files. However, you have more options to provide them to your builds, and those options have more precedence if you defined the same variable in your buildspec. Besides, all the options listed below allow you to define plain-text environment variables and reading parameters from the Parameter Store or Secrets Manager.

1) You can override environment variables while triggering your builds using AWS CLI or SDK using the environment-variables-override option, which has the highest precedence over other methods. Please see AWS CodeBuild CLI documentation on how to use them.

2) You can define environment variables while configuring your build project on AWS CodeBuild, and this option has higher precedence than the buildspec file.

CodeBuild Console environment variable definition

Therefore, the lowest precedence is your buildspec file if you also defined the same environment variable while triggering your builds or configuring your build project.

Conclusion

For some reason, you may need to use environment variables in your build projects. The reason may be to make them more readable and easily editable or use a central parameter and secrets store. In this post, I tried to explain how to reference them in your AWS CodeBuild buildspec files and export the ones exportable.

Thanks for reading!

Would you like to learn AWS CodePipeline and CodeBuild?

If you would like to learn how to use AWS CodePipeline with CodeBuild to create CI/CD pipelines to build your applications and deploy to EC2 and S3, you can join my AWS CodePipeline Step by Step course on Udemy with a special discount by clicking the link below.

https://www.udemy.com/course/aws-codepipeline-step-by-step/?couponCode=START-2021

Alternatively, use the coupon code START-2021 during the checkout.

Although we do not cover using environment variables with CodeBuild in this detail in the course, you can still learn how to use AWS CodeBuild with AWS CodePipeline along with other developer tools like CodeCommit and CodeDeploy to create pipelines on AWS. We make different examples with CodeBuild in the course.

Hope to see you in AWS CodePipeline Step by Step!

References

Emre Yilmaz

Independent AWS Consultant @ Shikisoft

AWS Certified Solutions Architect (Pro) • DevOps Engineer

Follow

Online Courses

Join my Udemy courses with a discount!

AWS CodePipeline Step by Step course logo AWS CodePipeline Step by Step course logo

AWS CodePipeline Step by Step

Learn how to create CI/CD pipelines using AWS CodePipeline, CodeCommit, CodeBuild, CodeDeploy, and CloudFormation. Automate your deployments to Amazon EC2 and S3.

Use the link below to enroll in this course with special discount.

Enroll Now on Udemy!

Alternatively, use START-2021 coupon code during the checkout.

AWS CloudFormation Step by Step: Beginner to Intermediate course logo AWS CloudFormation Step by Step: Beginner to Intermediate course logo

AWS CloudFormation Step by Step: Beginner to Intermediate

Learn how to manage your infrastructure as code on AWS. Start creating CloudFormation stacks and writing your own templates using YAML as a beginner step by step.

This course will also be a basis for its advanced level version in the upcoming months.

Use the link below to enroll in this course with special discount.

Enroll Now on Udemy!

Alternatively, use START-2021 coupon code during the checkout.

RSS

Subscribe to this blog's RSS feed

Categories