3 Ways to Schedule AWS Lambda and Step Functions State Machine Executions

  • by Emre Yilmaz
  • Jan 15, 2020
  • AWS, Serverless, DevOps
  • Istanbul
Scheduling AWS Lambda and Step Functions Executions

In addition to API development, AWS Lambda has many use cases. One of them is running some background jobs in scheduled intervals. Besides, if you need a chain of sequential or parallel AWS Lambda functions, the ideal way to orchestrate them is using AWS Step Functions.

In this post, I will talk about how to schedule your AWS Lambda functions or Step Functions state machine executions using AWS CloudWatch and EventBridge consoles as well as AWS Serverless Application Model (SAM) and CloudFormation templates.

What is a scheduled Amazon CloudWatch event?

As we discussed in our previous posts, Amazon CloudWatch is your monitoring service on AWS. It collects metrics and logs from your resources. Amazon CloudWatch Events is a sub-service of CloudWatch which streams near-real time system events upon changes on your AWS resources. You can take actions by defining targets to these events. For example, if one of your EC2 instances is started, it streams an event to Amazon CloudWatch Events. You can create a rule to trigger an AWS Lambda function whenever this event happens. This type of events are triggered when a change occurs on your AWS resources.

On the other hand, you can also define event rules that self-trigger regularly and configure a target action to do some regular work. So you can define an Amazon Lambda function or AWS Step Functions state machine as scheduled targets. Hence, when this event is triggered at the specified time or interval you defined, your function or state machine is executed. These types of events are called scheduled Amazon CloudWatch Events and we will make example of them in this blog post.

By the way, Amazon EventBridge is the new version of CloudWatch events, a serverless event bus service. It also allows you to build an event-driven architecture and integrate your SaaS applications with AWS resources easier. I will also show you how you can use Amazon EventBridge console to create scheduled events.

How to define a schedule for an event?

There are two options to define a scheduled event:

1) You can use a cron expression. This type of usage is very similar to the cron jobs you define on your Linux servers. But this time, it will be serverless and you will not need to pay for the idle time your servers run. For example, you can configure an event to be triggered on every Monday at 10:00 in GMT.

2) You can define a simple fixed interval, a rate expression, such as every 5 minutes, every 15 days, etc. You have less control in rate definitions. It starts from the time you create the event and calculates the next trigger time according to its creation time.

Scheduling an AWS Lambda function

Now let’s talk about how to schedule a single AWS Lambda function. I will show how to achieve this on Amazon CloudWatch Console first, then using the new Amazon EventBridge Console, and finally in your AWS SAM templates.

Using Amazon CloudWatch Console

Let’s start with Amazon CloudWatch Events Console to create a scheduled event for our AWS Lambda function. To do this,

  • click Rules under the Events section on the left menu. This will lead you to the Events list.
  • To create a new rule, click Create rule button at the top of this list.

Create rule button on Amazon CloudWatch Events

Defining a pattern for the rule

Next, you will see the event rule form in which Event pattern is selected. It is for the resource originated events that we discussed above. Hence, for scheduled events, you need to select Schedule option.

You will see that rate expression is enabled by default when you enable Schedule option. There are two information you need to provide to define a fixed rate: The interval and its unit. The unit can be minutes, hours and days. Hence, the minimum interval can be 1 minute. At this time, you cannot define an interval in seconds.

Scheduled Rate Expression on Amazon CloudWatch Events

This is how to provide a fixed rate expression. But let’s define a cron expression by selecting Cron expression option.

In this example, I will define an expression to trigger this event in every Tuesday at 10:00 AM in GMT+3 time zone. The result will be as below.

Scheduled Cron Expression on Amazon CloudWatch Events

Now let me explain what 0 7 ? * TUE * means briefly.

  • Firstly, we need to convert the time to GMT time zone. Because all schedules should be in this time zone. Therefore, 10:00 AM GMT+3 becomes 7:00 AM GMT. The first two character is minutes and hours (24-hour style) respectively, so we have 0 7 as a start.
  • The third character is the day of month in numbers. For example, if you use 4 here, it will trigger the event in every 4th of that month. But in that case, TUE value in the fifth character will conflict with this. In our example we don’t care which date it is, it should be just Thursday, so we use ? to express this.
  • The forth character is the month of the year. We want this event to be triggered every week regarless of the month it is in, so we use * to select all months.
  • The fifth character is the day of the week. We use TUE to select only Tuesdays from the week.
  • The last character is the year. We use * again not to provide a year filter and select all years.

You can learn more on Schedule Expressions for Rules reference. In addition to these, when you complete your rule, Amazon CloudWatch Console demonstrates dates and times of the next 10 events. It is pretty helpful to validate.

Defining AWS Lambda as target

Now we need to define our AWS Lambda function as target to this event in order to trigger it when the event happens. Click Add target button under the Targets section on the right.

Add Target button on Amazon CloudWatch event creation

By default, it will be a Lambda target, but if you click the list you can see that there are more options for this.

Target options on Amazon CloudWatch event creation

For this example, I created a simple AWS Lambda function named testFunction. So I select it from the list like below.

Lambda function as CloudWatch scheduled event target

You can select an alias/version or configure the input to transform it. But I will leave that part to you. You can see a sample for the events by clicking Show sample event below the cron expression.

Sample scheduled CloudWatch event

To continue creating your scheduled event, click Configure details button.

Configure Details

Now you need to provide a name and description to your event rule and click Create rule below it. Do not forget to leave Enabled as checked. Otherwise, you disable the rule and it will not be triggered. But of course, you can enable/disable a rule in the future whenever you want.

Scheduled CloudWatch event rule details

After these, your event rule will be created and you will be redirected to the Rules list again. Actually, this list displays not only the scheduled but all event rules you have. But we have only one rule as an example.

CloudWatch event rules list after creation

Now this event will be triggered in every Tuesdays at 7 AM in GMT and it will also trigger our AWS Lambda function accordingly.

Using Amazon EventBridge Console

Amazon EventBridge is the new version of Amazon CloudWatch Events service. It is a serverless event bus and also helps you to integrate your SaaS applications by streaming data into AWS and makes building event driven architectures easier. It can be your central place on AWS to manage all your events. So I will show you how to create a scheduled event rule on Amazon EventBridge Console.

Firstly, you need to go to Amazon Event Bridge console and click Rules from the left menu.

Amazon EventBridge Rules List

To create a new rule, click one of the Create rule buttons on the page. It will lead you to a form to create event rules. On this form, you need to provide a name and description to your rule like we did Configure details part on CloudWatch Console. Actually, Description is not mandatory, but it will be a good practice to write a meaningful explanation to remember it later easily.

Amazon EventBridge event rule creation - name and description

Next, you need to choose Schedule option on Define pattern section. The default Event pattern option is for defining triggers from resource based events.

Here you can provide a fixed rate schedule such as 30 minutes like below. If you do this, it will be triggered in every 30 minutes from the time you create.

Amazon EventBridge - Create Rule - Fixed Rate Schedule

But let’s make a cron example such as every Thursday at 7 AM GMT. Its cron expression becomes 0 7 ? * THU * and you can define it by selecting Cron expression option.

Amazon EventBridge - Create Rule - Cron Expression Schedule

This time we do not see the next 10 schedules as in AWS CloudWatch Events Console. It would be convenient if we had this. Maybe, in the future, they can build this feature to here as well.

Next, leave AWS default bus selected and proceed to the Targets section below to define your AWS Lambda function as target to this event. By default Lambda function option will be selected in the list, you will only need to select your AWS Lambda function from the function list.

Amazon EventBridge - Create Rule - Lambda Target

To finish the creation, click Create and you will be redirected back to the Rules list where you can see your new scheduled event.

Amazon EventBridge - Create Rule - Lambda Target

From now on, your AWS Lambda function will be triggered every Thursdays at 7 AM in GMT time zone. You can see your new scheduled rule on Amazon CloudWatch Events console as well.

Using AWS Serveress Application Model (SAM)

If you read my previous blog posts, I do not use AWS Management Console to define resources for production environments. I only use it for trying something, viewing or demonstrating resources. I mainly use AWS CloudFormation, and SAM templates for serverless resources. So I would like to demonstrate how you can configure scheduled events in AWS SAM as well.

Actually, it is very simple. You can achieve all we did above with a few lines of code as below.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  Sample SAM Template for triggering AWS Lambda functions via scheduled CloudWatch events.

Globals:
  Function:
    Timeout: 3
    Runtime: python3.7

Resources:
  TestFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: test_function/
      Handler: app.lambda_handler
      Events:
        ScheduledEvent:
          Type: Schedule
          Properties:
            Schedule: cron(0 7 ? * TUE *)
            Description: Sample Lambda function scheduling with SAM
            Enabled: True

It is repeatable and testable. The beauty of infrastructure as code is obvious.

AWS SAM has built-in support for scheduled CloudWatch events as it is one of the most common triggers of AWS Lambda. Let’s explain the crucial parts here.

  • The Type of your event should be Schedule.
  • Under the Properties of this event, you should have a Schedule property which takes the rule expression I explained above. Hence, cron(0 7 ? * TUE *) means every Tuesday at 7:00 AM GMT. But if you need to define a rate expression like in every 30 minutes, you can replace this with rate(30 minutes), too. You can customize the field between parantheses as you like once you decide whether you will use cron() or rate().
  • Description property is not mandatory, but I often use it as a best practice. There is also a Name attribute which you can give a name to your event, but I prefer to omit and leave it to CloudFormation to generate it from the stack name. If you use the same name for different events, you may end up with conflicts between your templates. So it is better to make it dynamic. If you still need to name your event, I recommend to use !Sub function and prepend the stack name to it.
  • Enabled attribute defines whether you enable or disable this scheduled event as we talked before while using Amazon CloudWatch Console.

This is how to schedule a single AWS Lambda function, now let’s talk about scheduling an AWS Step Functions state machine.

Scheduling an AWS Step Functions state machine execution

If you need to trigger a sequence of AWS Lambda functions or execute some of them in parallel, AWS Step Functions is the proper way to orchestrate this. The good thing is you can schedule AWS Step Functions - state machine executions like an AWS Lambda function. CloudWatch Events has built-in support for AWS Step Functions, too.

For this example, I created 2 AWS Lambda functions a simple AWS Step Functions State Machine to execute them sequentially. Here is the diagram of my state machine:

Step Functions state machine diagram for the example

Using Amazon CloudWatch Console

Again let’s start with Amazon CloudWatch Console. The rule expression definition will be same same as above, but the target part will be different. This time when you click Add target button, you need to select Step Functions state machine from the listbox instead of the default Lambda function

Selecting state machine option

When you do this, you will not see Lambda specific fields like alias/version as before. You only need to select the state machine that you would like to trigger as target from the list.

Selecting the state machine that will be executed as target

That’s all. The final screen will be as below, just click Configure details and give a name to your scheduled event as we did in AWS Lambda part.

CloudWatch Events rules list after scheduled lambda rule creation

Using Amazon EventBridge Console

Now let’s talk how to achieve this Amazon EventBridge Console. Again, only the target part will be changed on the assumption that you want to trigger your event in the same interval.

  • You should select Step Functions state machine instead of Lambda.
  • You should select your state machine from the list afterwards.

Selecting the state machine on EventBridge as target

As you see, there is also an IAM role field here. Your event should have privileges to execute this state machine in order to trigger it as a target. If you have an existing IAM role you can use it or leave it to EventBridge console to create a new one as I do here.

Using AWS CloudFormation and SAM

Unfortunately, AWS SAM has not built-in support for step functions or its scheduled event triggers yet. But this does not mean that we cannot define state machines in our SAM templates, actually we can. Because AWS SAM is based on AWS CloudFormation. Hence, we can define anything in a SAM template as long as it is supported by AWS CloudFormation.

But why use AWS SAM to define a Step Functions state machine? Your state machine will trigger one or more AWS Lambda functions which are very simple to define and organize with SAM. So I recommed to use AWS SAM for other benefits even though it does not have support for Step Functions yet.

Defining AWS Lambda functions and State Machine in AWS CloudFormation

I will not dive into the details of defining state machines in a CloudFormation template. If you know AWS CloudFormation well enough, it should be easily understandable for you. But let me explain a few things.

  • In the template below I have 2 AWS Lambda functions and a state machine that triggers them sequentially. It is same as the state machine I talked above.
  • You need to define an IAM role for your state machine that has privileges to execute your AWS Lambda functions. This one is different than the one I explained above.
  • You need to provide the Amazon States Language definitions for your state machine as a JSON string in the DefinitionString property. To create dynamic strings containing ARNs of your AWS Lambda functions, you can use the intrinsic Fn::Join function of AWS CloudFormation like I do below.
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  Sample SAM Template for triggering AWS Step Functions state machines via scheduled CloudWatch events.

Globals:
  Function:
    Timeout: 3
    Runtime: python3.7

Resources:

  # AWS Lambda functions ----
  TestFunction1:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: test_function1/
      Handler: app.lambda_handler

  TestFunction2:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: test_function2/
      Handler: app.lambda_handler

  # IAM role for the Step Functions State Machine
  StateMachineIAMRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          -
            Effect: "Allow"
            Principal:
              Service:
                  - Fn::Sub: "states.${AWS::Region}.amazonaws.com"
            Action:
              - "sts:AssumeRole"
      Policies:
        -
          PolicyName: StatesExecutionPolicy
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              -
                Effect: "Allow"
                Action: "lambda:InvokeFunction"
                Resource:
                  - !GetAtt TestFunction1.Arn
                  - !GetAtt TestFunction2.Arn

  # State Machine to orchestrate AWS Lambdas ---
  StateMachine:
    Type: AWS::StepFunctions::StateMachine
    Properties:
      StateMachineName: !Sub "${AWS::StackName}-StateMachine"
      RoleArn: !GetAtt StateMachineIAMRole.Arn
      DefinitionString: !Join
        - "\n"
        - - '{'
          - ' "StartAt": "TestFunction1",'
          - ' "States": {'
          - '   "TestFunction1": {'
          - '     "Type": "Task",'
          - !Sub '      "Resource": "${TestFunction1.Arn}",'
          - '     "Next": "TestFunction2"'
          - '   },' # end-of-TestFunction1
          - '   "TestFunction2": {'
          - '     "Type": "Task",'
          - !Sub '      "Resource": "${TestFunction2.Arn}",'
          - '     "End": true'
          - '   }' # end-of-TestFunction2
          - ' }' # end-of-States
          - '}' # end-of-Definitions 

This will create the state machine below, just like the one I showed you earlier.

State machine created with SAM

Now let’s define a scheduled event for this state machine.

Defining Scheduled CloudWatch Event

When compared to AWS Lambda and its scheduled event definitions, we have a few more work to do. We need to define 2 things:

  1. A second IAM role that will be assumed by AWS CloudWatch Events rule and have necessary privileges to trigger the state machine.
  2. A CloudWatch scheduled event that assumes this role and triggers the state machine above.

Hence, these lines will be added to our template.

...
  ScheduledEventIAMRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          -
            Effect: "Allow"
            Principal:
              Service:
                  - Fn::Sub: "events.amazonaws.com"
            Action:
              - "sts:AssumeRole"
      Policies:
        -
          PolicyName: StateMachineExecutionPolicy
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              -
                Effect: "Allow"
                Action: "states:StartExecution"
                Resource:
                  - !Ref StateMachine

  ScheduledEventRule:
    Type: "AWS::Events::Rule"
    Properties:
      Description: "Scheduled event to trigger Step Functions state machine"
      ScheduleExpression: cron(0 7 ? * TUE *)
      State: "ENABLED"
      Targets:
        -
          Arn: !Ref StateMachine
          Id: !GetAtt StateMachine.Name
          RoleArn: !GetAtt ScheduledEventIAMRole.Arn

Again let’s explain a few things:

  • Your event rule type should be AWS::Events::Rule.
  • This time you provide your schedule in ScheduleExpression property in the form of cron or rate expressions. This part is same as the Schedule property in our AWS Lambda scheduling example above.
  • There is no Enabled property. Instead, we have State property which takes either ENABLED or DISABLED.
  • You can give a Name as I explained in AWS Lambda example, but I left it to AWS CloudFormation.
  • Targets property is the place where you define your state machine as target. You use Arn property to do this. Id can be anything, but I use the name of the state machine to distinguish it easily. You also provide the IAM role we defined to let the event rule assume it while triggering the state machine using the RoleArn field.

Let me add that, you can define multiple targets to your scheduled events. But the aim of this example is doing them separately and explaining the differences in them.

Conclusion

In this post, we made an introduction to scheduled events to trigger AWS Lambda functions or AWS Step Functions state machines. Scheduling AWS Lambda function is good, but scheduling an AWS Step Functions state machine is great! Because you should use AWS Lambda functions as small pieces of your applications and if you need to do more, you should consider AWS Step Functions. We applied this to some of my clients effectively with the addition of retries, SNS notifications, etc. So I definitely recommend considering AWS Step Functions.

By the way, AWS Serverless Application Model is also becoming a thing. I see that it is evolving day by day. I am one of the early adopters of SAM because it is a subset of AWS CloudFormation.

Thanks for reading!

Would you like to learn CloudFormation?

By the way, if you are a beginner to AWS CloudFormation and would like to learn it in the right way, please check out my course on Udemy:

AWS CloudFormation Step By Step: Beginner to Intermediate

This course is a beginner level course on AWS CloudFormation teaching you how to create your own stacks, write your own templates using YAML, use parameters, conditions, mappings in them as well as how to use change set to update your stacks. It takes you to an intermediate level on CloudFormation and it will be a basis for my advanced level AWS CloudFormation course in the future.

The link above also includes a coupon for you to enroll in this course for only $12.99. Click the link or use JUNE2020 coupon code instead.

Hope to see you there!

References

Emre Yilmaz

Freelance AWS Consultant • AWS Certified Solutions Architect (Pro) • DevOps Engineer • Udemy Instructor

CEO @ Shikisoft

Follow

Online Courses
AWS CodePipeline Step by Step course logo AWS CodePipeline Step by Step course logo

AWS CodePipeline Step by Step

Learn how to create CI/CD pipelines using AWS CodePipeline, CodeCommit, CodeBuild, CodeDeploy, and CloudFormation. Automate your deployments to Amazon EC2 and S3.

Use the link below to enroll in this course with up to 85% discount.

Enroll Now on Udemy!

Alternatively, use AUG2020 coupon code during the checkout.

AWS CloudFormation Step by Step: Beginner to Intermediate course logo AWS CloudFormation Step by Step: Beginner to Intermediate course logo

AWS CloudFormation Step by Step: Beginner to Intermediate

Learn how to manage your infrastructure as code on AWS. Start creating CloudFormation stacks and writing your own templates using YAML as a beginner step by step.

This course will also be a basis for its advanced level version in the upcoming months.

Use the link below to enroll in this course with up to 85% discount.

Enroll Now on Udemy!

Alternatively, use AUG2020 coupon code during the checkout.

RSS

Subscribe to this blog's RSS feed

Categories