AWS Step Functions Nested Workflows
What Is a Nested Workflow?
Simply put a nested workflow is a workflow within a workflow... workflow-ception! It is a feature requested of AWS Step Functions for a long time now and it was recently released (read the blog post: here). The core problem users were finding with Step Functions was they had no way of re-using workflows, if you had two workflows A and B and B needed to do everything A did and a little more it would make sense for B to use A and then do it's little bit afterwards, right? Well that wasn't possible (without a lot of scrappy workarounds) until now.
In this article I am going to dive in to the new nested workflows feature.This article assumes you are reasonably comfortable with AWS Step Functions and know the foundations of how to use it. Calling nested workflows is done using an AWS Step Functions integration state that specifies the StateMachineArn of the state machine to execute.
If you want a quick refresher on AWS Step Functions then I recommend the video below. After you watch this video you should have the basic knowledge required for this article.
I also highly recommend reading the AWS Step Functions developer guide which can be found here.
Asynchronous Nested Workflows
If you don't wait to wait for a nested workflow to finish before continuing your own then you can call a nested workflow asynchronously and proceed with the rest of the parent workflow.
This is perfect for any workflows that you want to run in the background while another parents workflow runs, for example you could have a parent workflow for uploading images and a child workflow for resizing the image and storing it in different formats.
An example state that starts a state machine and immediately move on is given below.
Synchronous Nested Workflows
If you need to wait for a nested state machine to finish executing before proceeding with the parent state machine then you need to start the nested state machine synchronously. There are two ways of doing this, right now we will cover the first.
The call is essentially completely the same as the asynchronous call except we add in .sync to the end of the Resource value. This indicates to Step Functions to pause the current execution of the parent workflow until the child workflow either succeeds or fails.
The .sync method is also the only method where the output of the child state machine execution is automatically returned to the parent state machine.
Synchronous Nested Workflows with Task Tokens
This is the second, more in-depth, way of performing a synchronous nested workflow call. It uses an already established pattern within AWS Step Functions known as the callback service integration pattern. In short this pattern allows you to issue a task token to a service you are calling and then pauses the execution of the calling state machine until a call to SendTaskSuccess or SendTaskFailure is made using the task token by the child service.
There are two main advantages to this pattern over the previous .sync method:
- Heartbeats: You can use the task token to call SendTaskHeartbeat allowing your child workflow to indicate to the parent workflow it is still running. This allows you to ensure your parent workflow is never halted on a stuck child workflow by setting a HeartbeatSeconds timeout value. For example if HeartbeatSeconds is 30 then you expect a heartbeat call from the child workflow every 30 seconds or else you declare the execution a failure.
- You don't have to wait for the whole child: Any state in the child workflow can call SendTaskSuccess or SendTaskFailure, meaning if for example you only need to reach state 3 out of 5 in the child workflow before restarting the parent workflow you can do exactly that. Alternatively your last step can make the call, effectively emulating the .sync method.
You can see below the special task token value being passed as input to the child workflow as part of the context object (specified by $$). See here for how to access the context object.
Tracking the Parent Workflow
Step Functions has added a new special parameter to allow you to track what workflow started the execution of another workflow, this parameter is called AWS_STEP_FUNCTIONS_STARTED_BY_EXECUTION_ID.
This special parameter can be accessed in the child workflow that gets started by accessing the special context object that Step Functions exposes.
- AWS Step Functions Context Object
The AWS Step Functions context object allows you to access metadata about the state machine and state executions from within. The Map state also exposes further special properties.
Unfortunately the output of a nested workflow is presented in the form of escaped JSON, this presents an issue because you cannot naively use the JSON in your next state. For example if your next state is a Choice state that tries to access the output of the nested workflow it will fail because the JSON is escaped.
My current recommendation is to have a Lambda function that simply un-escapes the JSON and outputs it, placing this Lambda after each nested workflow to ensure the JSON is always in the correct format.
Hopefully AWS take note of people having a hard time with the escaped JSON and change how the output of nested workflows in handled, but for now a simple workaround with a Lambda isn't too hard to do.
Another New State: Map
AWS Step Functions recently got upgraded with another state, the Map state which allows for dynamic parallelism for array processing. The article below discusses the new state and it's special attributes.
This new Map state would allow you to apply your nested workflow to an array of items with parallelism controlled by you. The combination of nested workflows with the Map state has made Step Functions significantly more user friendly and powerful.
- AWS Step Function Map State
AWS Step Functions has released support for dynamic parallelism in the form of the new Map state. This article takes a look at the new state and it's special parameters and properties.