Defining Environment Variables in AWS CodeBuild Buildspec Files

  • by Emre Yilmaz
  • Jan 7, 2021
  • AWSDevOps
  • Istanbul
Defining Environment Variables in CodeBuild Buildspec Files

AWS CodeBuild is the serverless build service of AWS, and until now, I covered various ways of using it in my previous posts. Today, I would like to talk about how to define and use environment variables in your build specification files or, in other words, buildspecs. In a build project, you can assign plain texts to your environment variables, read parameters from AWS Systems Manager Parameter Store, or retrieve secrets from AWS Secrets Manager. I will give examples to each.

Besides, you can also use AWS CodeBuild with AWS CodePipeline as a part of a CI/CD pipeline. We discuss how to do this in my AWS CodePipeline Step by Step course. Therefore, I will also give an example of exporting an environment variable from a build to be able to use it in a later pipeline action.

Why use environment variables in a build project?

AWS CodeBuild works like a terminal in your pipeline. You provide your code as an input artifact, for example, as a source artifact from a CodeCommit repository, and run some commands on it. These commands may be part of a build process or a test action. Therefore, let’s list why you may need to use environment variables in your builds briefly.

  • The first apparent reason is simplifying your build commands and managing your buildspec file efficiently. You can define environment variables as key-value pairs and refer to them in your commands.
  • You may need to load some configuration parameters from AWS Systems Manager Parameter Store before your build commands run.
  • Or, you can even load sensitive parameter values like usernames and passwords from AWS Secrets Manager.

AWS CodeBuild buildspecs support all these options. So next, let me show you examples of them.

Defining a literal environment variable

In a buildspec file, you define environment variables under the env section. You place your simple environment variables by defining the variables section in env. In the example below, we define an S3_BUCKET variable and assign “my-website-bucket” as its value. Then, we reference this value in the build phase, just like a regular environment variable using the dollar ($) sign.

version: 0.2
env:
  variables:
    S3_BUCKET: "my-website-bucket"
phases:
  ...
  build:
    commands:
      - echo "S3 bucket is $S3_BUCKET"
  ...
artifacts:
    ...

The crucial thing you should note here is that you can only assign literal values to the environment variables declared this way. You cannot assign dynamic values at runtime. If you would like to change the value of the S3_BUCKET variable above, you have to change your buildspec file and push your changes to your repository again. So it is like hardcoding parameter values. But it is better than typing the S3 bucket name in all commands needed in the phases section.

Getting environment variable values from AWS Systems Manager Parameter Store

AWS Systems Manager Parameter Store allows you to store your configuration parameters or secrets securely and scalably. It is an AWS-managed, centralized parameter storage. You can both store encrypted or unencrypted parameters in it. You can even define your parameters under a hierarchy such as an organization. Hence, the Parameter Store has many useful features.

For CodeBuild usage, you can define your parameters on Parameter Store, assign them to an environment variable one by one in your buildspec. Then your build containers can load them before executing your commands. To do this, you use the parameter-store section under env and provide key-value pairs where the key would be the local environment variable used by your commands. The value would be the parameter key name on AWS Systems Manager Parameter Store.

For example, in the code below, we read values for a simple my_s3_bucket parameter, a /my_org/s3_bucket parameter which is actually an s3_bucket parameter under /my_org label, and an encrypted db_username parameter from the Systems Manager Parameter Store.

version: 0.2
env:
  parameter-store:
    S3_BUCKET: "my_s3_bucket"
    ORG_BUCKET: "/my_org/s3_bucket"
    DB_USERNAME: "db_username"
  ...

Also, the sample parameters retrieved here would be like below on AWS Systems Manager Parameter Store Console.

Sample parameters on AWS Systems Manager Parameter Store

The IAM service role assigned to your CodeBuild project must have ssm:GetParameters permissions to access these parameters. Their values will be assigned to S3_BUCKET, ORG_BUCKET, and DB_USERNAME environment variables. You can refer them using a dollar sign ($) like $ORG_BUCKET, as you would do while accessing a regular environment variable in your commands.

Reading values from AWS Secrets Manager

AWS Secrets Manager is a central, encrypted secrets manager provided by AWS. Although you can use the Systems Manager Parameter Store to store encrypted values, Secrets Manager offers more options such as seamless integration with Amazon RDS, DocumentDB, Redshift databases, automatic secrets rotation, etc. But it is not free, unlike the Parameter Store.

You define secrets as key-value pairs in JSON format in AWS Secrets Manager. You can store multiple key-value pairs in a single secret. So the access method from a CodeBuild buildspec follows the same pattern.

This time, you define your environment variables under the secrets-manager key inside the env section. Again your key will be the local environment variable which the secret value will be loaded into, and its value will be in secret-name:JSON-key format.

For example, in the buildspec below, we access the username and password keys defined in the my_secret secret on AWS Secrets Manager.

version: 0.2
env:
  secrets-manager:
    DB_PASSWORD: "my_secret:password"
    DB_USER: "my_secret:username"
  ...

The sample secret definition on AWS Secrets Manager Console:

Sample secret on AWS Secrets Manager

The value of the sample secret in the form of key-value pairs:

Sample secret key-value pairs on AWS Secrets Manager

The value of the sample secret in JSON format on AWS Secrets Manager Console:

Sample secret value in JSON on AWS Secrets Manager

As our previous methods, you can reference these values using $DB_PASSWORD and $DB_USER environment variables. But the IAM service role attached to your CodeBuild project must have secretsmanager:GetSecretValue permissions for your secret, which is my_secret in our example.

Environment variables provided by CodeBuild

AWS CodeBuild provides some environment variables specific to the build that is in progress, such as CODEBUILD_BUILD_ID and CODEBUILD_BUILD_NUMBER during the build execution. You can use them for logging purposes or export them if you have other needs in your pipelines. Please find the variables provided by CodeBuild on AWS CodeBuild - Environment Variables Reference.

Exporting variables from a build project

You can export your literal environment variables in your buildspec files to make them reusable in later CodePipeline action configurations but not the ones read from AWS Systems Manager Parameter Store or AWS Secrets Manager. Also, you cannot export environment variables starting with AWS_.

To do this, you need to provide their variable names as a list under env/exported-variables. For example, the buildspec file below exports the custom S3_BUCKET and CodeBuild-specific CODEBUILD_BUILD_ID environment variables.

version: 0.2
env:
  variables:
    S3_BUCKET: "my-website-bucket"
  exported-variables:
    - S3_BUCKET
    - CODEBUILD_BUILD_ID
  ...
artifacts:
    ...

After the CodeBuild project runs, your CodePipeline action history will show them in action details like below.

Build action output on AWS CodePipeline

You will need to assign an output variable namespace in your build action to reference these variables in later CodePipeline actions in the form of #{VariableNameSpace.VariableName}. If you finished my AWS CodePipeline Step by Step course, you would remember our lecture about using action variables on CodePipeline with a CloudFormation staging stack output and manual approval action configuration example. Although the example in this post is different, the usage is similar.

Other options to define environment variables and precedence

In this post, I concentrated on defining environment variables in CodeBuild buildspec files. However, you have more options to provide them to your builds, and those options have more precedence if you defined the same variable in your buildspec. Besides, all the options listed below allow you to define plain-text environment variables and reading parameters from the Parameter Store or Secrets Manager.

1) You can override environment variables while triggering your builds using AWS CLI or SDK using the environment-variables-override option, which has the highest precedence over other methods. Please see AWS CodeBuild CLI documentation on how to use them.

2) You can define environment variables while configuring your build project on AWS CodeBuild, and this option has higher precedence than the buildspec file.

CodeBuild Console environment variable definition

Therefore, the lowest precedence is your buildspec file if you also defined the same environment variable while triggering your builds or configuring your build project.

Conclusion

For some reason, you may need to use environment variables in your build projects. The reason may be to make them more readable and easily editable or use a central parameter and secrets store. In this post, I tried to explain how to reference them in your AWS CodeBuild buildspec files and export the ones exportable.

Thanks for reading!

Would you like to learn AWS CodeBuild With AWS CodePipeline?

If you would like to learn to use AWS CodeBuild with AWS CodePipeline to create CI/CD pipelines to build your applications and deploy them to Amazon EC2 and S3, you can join my AWS CodePipeline Step by Step course on Udemy.

Although we do not cover using environment variables on AWS CodeBuild projects in this detail, you can still learn to use AWS CodePipeline with AWS CodeBuild and developer tools like AWS CodeCommit and AWS CodeDeploy to create pipelines on AWS. We make different examples with AWS CodeBuild in the course.

This link also includes a special coupon for you. Feel free to use it if Udemy does not have a better promotion at this time.

Besides, for all my available courses, please check out our Online Courses page. Hope to see you in them!

References

Emre Yilmaz

Independent AWS Consultant & Founder @ Shikisoft

Follow

Courses

View Shikisoft's online courses

Join Our AWS Courses!

Would you like to learn AWS services like AWS CodePipeline and AWS CloudFormation in detail with hands-on examples?

Find more details on our available courses and enroll in them with a discount!

View Our Online Courses

RSS

Subscribe to this blog's RSS feed

Categories