AWS Step Functions Nested Workflows

Updated on October 4, 2019
An example of a basic AWS Step Functions state machine.
An example of a basic AWS Step Functions state machine. | Source

What Is a Nested Workflow?

Simply put a nested workflow is a workflow within a workflow... workflow-ception! It is a feature requested of AWS Step Functions for a long time now and it was recently released (read the blog post: here). The core problem users were finding with Step Functions was they had no way of re-using workflows, if you had two workflows A and B and B needed to do everything A did and a little more it would make sense for B to use A and then do it's little bit afterwards, right? Well that wasn't possible (without a lot of scrappy workarounds) until now.

In this article I am going to dive in to the new nested workflows feature.This article assumes you are reasonably comfortable with AWS Step Functions and know the foundations of how to use it. Calling nested workflows is done using an AWS Step Functions integration state that specifies the StateMachineArn of the state machine to execute.

Nested workflows allow you to build larger workflows out of smaller workflows, promoting reuse and maintainability!
Nested workflows allow you to build larger workflows out of smaller workflows, promoting reuse and maintainability!

If you want a quick refresher on AWS Step Functions then I recommend the video below. After you watch this video you should have the basic knowledge required for this article.

I also highly recommend reading the AWS Step Functions developer guide which can be found here.

Asynchronous Nested Workflows

If you don't wait to wait for a nested workflow to finish before continuing your own then you can call a nested workflow asynchronously and proceed with the rest of the parent workflow.

Asynchronous nested workflow calls do NOT wait for the nested workflow to finish before moving on.
Asynchronous nested workflow calls do NOT wait for the nested workflow to finish before moving on.

This is perfect for any workflows that you want to run in the background while another parents workflow runs, for example you could have a parent workflow for uploading images and a child workflow for resizing the image and storing it in different formats.

An example state that starts a state machine and immediately move on is given below.

This state is calling a state machine and immediately moving on, it doesn't wait for the nested state machine to finish executing.
This state is calling a state machine and immediately moving on, it doesn't wait for the nested state machine to finish executing. | Source

Synchronous Nested Workflows

If you need to wait for a nested state machine to finish executing before proceeding with the parent state machine then you need to start the nested state machine synchronously. There are two ways of doing this, right now we will cover the first.

Synchronous nested workflows force the calling workflow to wait for their execution to finish before continuing.
Synchronous nested workflows force the calling workflow to wait for their execution to finish before continuing.

The call is essentially completely the same as the asynchronous call except we add in .sync to the end of the Resource value. This indicates to Step Functions to pause the current execution of the parent workflow until the child workflow either succeeds or fails.

The .sync method is also the only method where the output of the child state machine execution is automatically returned to the parent state machine.

This state starts a nested state machine and waits for it to finish, note the '.sync' to specify to wait for it to finish.
This state starts a nested state machine and waits for it to finish, note the '.sync' to specify to wait for it to finish. | Source

Synchronous Nested Workflows with Task Tokens

This is the second, more in-depth, way of performing a synchronous nested workflow call. It uses an already established pattern within AWS Step Functions known as the callback service integration pattern. In short this pattern allows you to issue a task token to a service you are calling and then pauses the execution of the calling state machine until a call to SendTaskSuccess or SendTaskFailure is made using the task token by the child service.

Using the callback token allows you to tell the calling workflow to continue anytime you want as well as allowing you to send heartbeats.
Using the callback token allows you to tell the calling workflow to continue anytime you want as well as allowing you to send heartbeats.

There are two main advantages to this pattern over the previous .sync method:

  • Heartbeats: You can use the task token to call SendTaskHeartbeat allowing your child workflow to indicate to the parent workflow it is still running. This allows you to ensure your parent workflow is never halted on a stuck child workflow by setting a HeartbeatSeconds timeout value. For example if HeartbeatSeconds is 30 then you expect a heartbeat call from the child workflow every 30 seconds or else you declare the execution a failure.
  • You don't have to wait for the whole child: Any state in the child workflow can call SendTaskSuccess or SendTaskFailure, meaning if for example you only need to reach state 3 out of 5 in the child workflow before restarting the parent workflow you can do exactly that. Alternatively your last step can make the call, effectively emulating the .sync method.

You can see below the special task token value being passed as input to the child workflow as part of the context object (specified by $$). See here for how to access the context object.

This task is calling a nested workflow and will wait until the task token is used to call SendTaskSuccess or SendTaskFailure.
This task is calling a nested workflow and will wait until the task token is used to call SendTaskSuccess or SendTaskFailure. | Source

Tracking the Parent Workflow

Step Functions has added a new special parameter to allow you to track what workflow started the execution of another workflow, this parameter is called AWS_STEP_FUNCTIONS_STARTED_BY_EXECUTION_ID.

This special parameter can be accessed in the child workflow that gets started by accessing the special context object that Step Functions exposes.

An example of the special execution ID variable being passed to the nested workflow being initiated.
An example of the special execution ID variable being passed to the nested workflow being initiated. | Source

Escaped Output

Unfortunately the output of a nested workflow is presented in the form of escaped JSON, this presents an issue because you cannot naively use the JSON in your next state. For example if your next state is a Choice state that tries to access the output of the nested workflow it will fail because the JSON is escaped.

My current recommendation is to have a Lambda function that simply un-escapes the JSON and outputs it, placing this Lambda after each nested workflow to ensure the JSON is always in the correct format.

Hopefully AWS take note of people having a hard time with the escaped JSON and change how the output of nested workflows in handled, but for now a simple workaround with a Lambda isn't too hard to do.

Another New State: Map

AWS Step Functions recently got upgraded with another state, the Map state which allows for dynamic parallelism for array processing. The article below discusses the new state and it's special attributes.

This new Map state would allow you to apply your nested workflow to an array of items with parallelism controlled by you. The combination of nested workflows with the Map state has made Step Functions significantly more user friendly and powerful.

Questions & Answers

    Comments

      0 of 8192 characters used
      Post Comment

      No comments yet.

      working

      This website uses cookies

      As a user in the EEA, your approval is needed on a few things. To provide a better website experience, owlcation.com uses cookies (and other similar technologies) and may collect, process, and share personal data. Please choose which areas of our service you consent to our doing so.

      For more information on managing or withdrawing consents and how we handle data, visit our Privacy Policy at: https://owlcation.com/privacy-policy#gdpr

      Show Details
      Necessary
      HubPages Device IDThis is used to identify particular browsers or devices when the access the service, and is used for security reasons.
      LoginThis is necessary to sign in to the HubPages Service.
      Google RecaptchaThis is used to prevent bots and spam. (Privacy Policy)
      AkismetThis is used to detect comment spam. (Privacy Policy)
      HubPages Google AnalyticsThis is used to provide data on traffic to our website, all personally identifyable data is anonymized. (Privacy Policy)
      HubPages Traffic PixelThis is used to collect data on traffic to articles and other pages on our site. Unless you are signed in to a HubPages account, all personally identifiable information is anonymized.
      Amazon Web ServicesThis is a cloud services platform that we used to host our service. (Privacy Policy)
      CloudflareThis is a cloud CDN service that we use to efficiently deliver files required for our service to operate such as javascript, cascading style sheets, images, and videos. (Privacy Policy)
      Google Hosted LibrariesJavascript software libraries such as jQuery are loaded at endpoints on the googleapis.com or gstatic.com domains, for performance and efficiency reasons. (Privacy Policy)
      Features
      Google Custom SearchThis is feature allows you to search the site. (Privacy Policy)
      Google MapsSome articles have Google Maps embedded in them. (Privacy Policy)
      Google ChartsThis is used to display charts and graphs on articles and the author center. (Privacy Policy)
      Google AdSense Host APIThis service allows you to sign up for or associate a Google AdSense account with HubPages, so that you can earn money from ads on your articles. No data is shared unless you engage with this feature. (Privacy Policy)
      Google YouTubeSome articles have YouTube videos embedded in them. (Privacy Policy)
      VimeoSome articles have Vimeo videos embedded in them. (Privacy Policy)
      PaypalThis is used for a registered author who enrolls in the HubPages Earnings program and requests to be paid via PayPal. No data is shared with Paypal unless you engage with this feature. (Privacy Policy)
      Facebook LoginYou can use this to streamline signing up for, or signing in to your Hubpages account. No data is shared with Facebook unless you engage with this feature. (Privacy Policy)
      MavenThis supports the Maven widget and search functionality. (Privacy Policy)
      Marketing
      Google AdSenseThis is an ad network. (Privacy Policy)
      Google DoubleClickGoogle provides ad serving technology and runs an ad network. (Privacy Policy)
      Index ExchangeThis is an ad network. (Privacy Policy)
      SovrnThis is an ad network. (Privacy Policy)
      Facebook AdsThis is an ad network. (Privacy Policy)
      Amazon Unified Ad MarketplaceThis is an ad network. (Privacy Policy)
      AppNexusThis is an ad network. (Privacy Policy)
      OpenxThis is an ad network. (Privacy Policy)
      Rubicon ProjectThis is an ad network. (Privacy Policy)
      TripleLiftThis is an ad network. (Privacy Policy)
      Say MediaWe partner with Say Media to deliver ad campaigns on our sites. (Privacy Policy)
      Remarketing PixelsWe may use remarketing pixels from advertising networks such as Google AdWords, Bing Ads, and Facebook in order to advertise the HubPages Service to people that have visited our sites.
      Conversion Tracking PixelsWe may use conversion tracking pixels from advertising networks such as Google AdWords, Bing Ads, and Facebook in order to identify when an advertisement has successfully resulted in the desired action, such as signing up for the HubPages Service or publishing an article on the HubPages Service.
      Statistics
      Author Google AnalyticsThis is used to provide traffic data and reports to the authors of articles on the HubPages Service. (Privacy Policy)
      ComscoreComScore is a media measurement and analytics company providing marketing data and analytics to enterprises, media and advertising agencies, and publishers. Non-consent will result in ComScore only processing obfuscated personal data. (Privacy Policy)
      Amazon Tracking PixelSome articles display amazon products as part of the Amazon Affiliate program, this pixel provides traffic statistics for those products (Privacy Policy)
      ClickscoThis is a data management platform studying reader behavior (Privacy Policy)