AWS CodeBuild is the serverless build service of AWS, and until now, I covered various ways of using it in my previous posts. Today, I would like to talk about how to define and use environment variables in your build specification files or, in other words, buildspecs. In a build project, you can assign plain texts to your environment variables, read parameters from AWS Systems Manager Parameter Store, or retrieve secrets from AWS Secrets Manager. I will give examples to each.
Besides, you can also use AWS CodeBuild with AWS CodePipeline as a part of a CI/CD pipeline. We discuss how to do this in my AWS CodePipeline Step by Step course. Therefore, I will also give an example of exporting an environment variable from a build to be able to use it in a later pipeline action.
Why use environment variables in a build project?
AWS CodeBuild works like a terminal in your pipeline. You provide your code as an input artifact, for example, as a source artifact from a CodeCommit repository, and run some commands on it. These commands may be part of a build process or a test action. Therefore, let’s list why you may need to use environment variables in your builds briefly.
- The first apparent reason is simplifying your build commands and managing your buildspec file efficiently. You can define environment variables as key-value pairs and refer to them in your commands.
- You may need to load some configuration parameters from AWS Systems Manager Parameter Store before your build commands run.
- Or, you can even load sensitive parameter values like usernames and passwords from AWS Secrets Manager.
AWS CodeBuild buildspecs support all these options. So next, let me show you examples of them.
Defining a literal environment variable
In a buildspec file, you define environment variables under the env section. You place your simple environment variables by defining the variables section in env. In the example below, we define an S3_BUCKET variable and assign “my-website-bucket” as its value. Then, we reference this value in the build phase, just like a regular environment variable using the dollar ($) sign.
version: 0.2
env:
variables:
S3_BUCKET: "my-website-bucket"
phases:
...
build:
commands:
- echo "S3 bucket is $S3_BUCKET"
...
artifacts:
...The crucial thing you should note here is that you can only assign literal values to the environment variables declared this way. You cannot assign dynamic values at runtime. If you would like to change the value of the S3_BUCKET variable above, you have to change your buildspec file and push your changes to your repository again. So it is like hardcoding parameter values. But it is better than typing the S3 bucket name in all commands needed in the phases section.
Getting environment variable values from AWS Systems Manager Parameter Store
AWS Systems Manager Parameter Store allows you to store your configuration parameters or secrets securely and scalably. It is an AWS-managed, centralized parameter storage. You can both store encrypted or unencrypted parameters in it. You can even define your parameters under a hierarchy such as an organization. Hence, the Parameter Store has many useful features.
For CodeBuild usage, you can define your parameters on Parameter Store, assign them to an environment variable one by one in your buildspec. Then your build containers can load them before executing your commands. To do this, you use the parameter-store section under env and provide key-value pairs where the key would be the local environment variable used by your commands. The value would be the parameter key name on AWS Systems Manager Parameter Store.
For example, in the code below, we read values for a simple my_s3_bucket parameter, a /my_org/s3_bucket parameter which is actually an s3_bucket parameter under /my_org label, and an encrypted db_username parameter from the Systems Manager Parameter Store.
version: 0.2
env:
parameter-store:
S3_BUCKET: "my_s3_bucket"
ORG_BUCKET: "/my_org/s3_bucket"
DB_USERNAME: "db_username"
...Also, the sample parameters retrieved here would be like below on AWS Systems Manager Parameter Store Console.
The IAM service role assigned to your CodeBuild project must have ssm:GetParameters permissions to access these parameters. Their values will be assigned to S3_BUCKET, ORG_BUCKET, and DB_USERNAME environment variables. You can refer them using a dollar sign ($) like $ORG_BUCKET, as you would do while accessing a regular environment variable in your commands.
Reading values from AWS Secrets Manager
AWS Secrets Manager is a central, encrypted secrets manager provided by AWS. Although you can use the Systems Manager Parameter Store to store encrypted values, Secrets Manager offers more options such as seamless integration with Amazon RDS, DocumentDB, Redshift databases, automatic secrets rotation, etc. But it is not free, unlike the Parameter Store.
You define secrets as key-value pairs in JSON format in AWS Secrets Manager. You can store multiple key-value pairs in a single secret. So the access method from a CodeBuild buildspec follows the same pattern.
This time, you define your environment variables under the secrets-manager key inside the env section. Again your key will be the local environment variable which the secret value will be loaded into, and its value will be in secret-name:JSON-key format.
For example, in the buildspec below, we access the username and password keys defined in the my_secret secret on AWS Secrets Manager.
version: 0.2
env:
secrets-manager:
DB_PASSWORD: "my_secret:password"
DB_USER: "my_secret:username"
...The sample secret definition on AWS Secrets Manager Console:
The value of the sample secret in the form of key-value pairs:
The value of the sample secret in JSON format on AWS Secrets Manager Console:
As our previous methods, you can reference these values using $DB_PASSWORD and $DB_USER environment variables. But the IAM service role attached to your CodeBuild project must have secretsmanager:GetSecretValue permissions for your secret, which is my_secret in our example.
Environment variables provided by CodeBuild
AWS CodeBuild provides some environment variables specific to the build that is in progress, such as CODEBUILD_BUILD_ID and CODEBUILD_BUILD_NUMBER during the build execution. You can use them for logging purposes or export them if you have other needs in your pipelines. Please find the variables provided by CodeBuild on AWS CodeBuild - Environment Variables Reference.
Exporting variables from a build project
You can export your literal environment variables in your buildspec files to make them reusable in later CodePipeline action configurations but not the ones read from AWS Systems Manager Parameter Store or AWS Secrets Manager. Also, you cannot export environment variables starting with AWS_.
To do this, you need to provide their variable names as a list under env/exported-variables. For example, the buildspec file below exports the custom S3_BUCKET and CodeBuild-specific CODEBUILD_BUILD_ID environment variables.
version: 0.2
env:
variables:
S3_BUCKET: "my-website-bucket"
exported-variables:
- S3_BUCKET
- CODEBUILD_BUILD_ID
...
artifacts:
...After the CodeBuild project runs, your CodePipeline action history will show them in action details like below.
You will need to assign an output variable namespace in your build action to reference these variables in later CodePipeline actions in the form of #{VariableNameSpace.VariableName}. If you finished my AWS CodePipeline Step by Step course, you would remember our lecture about using action variables on CodePipeline with a CloudFormation staging stack output and manual approval action configuration example. Although the example in this post is different, the usage is similar.
Other options to define environment variables and precedence
In this post, I concentrated on defining environment variables in CodeBuild buildspec files. However, you have more options to provide them to your builds, and those options have more precedence if you defined the same variable in your buildspec. Besides, all the options listed below allow you to define plain-text environment variables and reading parameters from the Parameter Store or Secrets Manager.
1) You can override environment variables while triggering your builds using AWS CLI or SDK using the environment-variables-override option, which has the highest precedence over other methods. Please see AWS CodeBuild CLI documentation on how to use them.
2) You can define environment variables while configuring your build project on AWS CodeBuild, and this option has higher precedence than the buildspec file.
Therefore, the lowest precedence is your buildspec file if you also defined the same environment variable while triggering your builds or configuring your build project.
Would you like to learn AWS CodeBuild With AWS CodePipeline?
You can find hands on lectures about defining environment variables on AWS CodeBuild projects on my AWS CodePipeline Step by Step course on Udemy. In our section about building Docker images and deploying them to Amazon ECS with CodePipeline, we have examples of storing Docker Hub credentials on AWS Secrets Manager or SSM Parameter Store and accessing them as environment variables from AWS CodeBuild.
Besides, you can learn to use AWS CodeBuild with AWS CodePipeline along with AWS CodeCommit and AWS CodeDeploy to create CI/CD pipelines to build your application code or Docker images, and automatically deploy them to Amazon EC2, S3 and Amazon ECS.
Use this link to get a special discount for this blog: Join AWS CodePipeline Step by Step on Udemy!
For all my available courses, please also check out our Online Courses.
Conclusion
You often need to use environment variables in your build projects on AWS CodeBuild. The reason may be to make your buildspecs more readable and easily editable, or use a central parameter and secrets stor for your configuration parameters or credentials. So, in this post, I tried to guide you on how to reference them in your AWS CodeBuild buildspec files and export the ones that are exportable.
Thanks for reading!
