AWS Step Function Map State - Owlcation - Education
Updated date:

AWS Step Function Map State

Author:
Dynamic parallelism has arrived for AWS Step Functions

Dynamic parallelism has arrived for AWS Step Functions

AWS Step Functions has received another upgrade, with dynamic parallelism being added in the form of the new Map state. This new state allows you to apply a state machine defined as an Iterator to an array of input objects with executions running in parallel as controlled by the value of the MaxConcurrency value.

This means there is no longer a need for the many workarounds that exist to achieve loops and parallelism, with the new Map state for-loop style executions can be achieved with a single state.

Think of the new Map state like a Java 8 stream map operator, where the Iterator you define is the function applied to each item in the stream.

If you need a refresher on what AWS Step Functions is then I recommend giving the video below a watch as it serves as a comprehensive overview.

Think of the new Map state like a Java 8 stream map operator, where the Iterator you define is the function applied to each item in the stream.

MaxConcurrency

The MaxConcurrency parameter for a Map state defines how many concurrent executions of the Iterator you define are allowed to run in parallel.

A value of 0 means the concurrency is unbounded and Step Functions will try and run as many concurrent executions as possible.

A value of 1 means there is no true concurrency, the array you pass will be processed sequentially such that the execution of the first item must completely finish before the execution of the second item can start.

Any value greater than 1 defines a limit for the maximum number of concurrent executions.

For values greater than 1 there is no guarantee the number of concurrent executions will always be the same as the maximum concurrency, but it will never be more.

Special Context Object Values

In the Map state the context object has two additional special values:

  • $$.Map.Item.Index: Defines the index in the array the item is found at.
  • $$.Map.Item.Value: Defines the JSON value of the array item being processed.
  • AWS Step Functions Context Object
    The AWS Step Functions context object allows you to access metadata about the state machine and state executions from within. The Map state also exposes further special properties.

Parameters

The parameters field can be used to customise the input to the iterator for each individual execution. For example the below JSON passes the deliveryAddress to each execution as well as passing the special context object values discussed previously.

"Parameters": {
  "index.$": "$$.Map.Item.Index",
  "item.$": "$$.Map.Item.Value",
  "deliveryAddress.$": "$.deliveryAddress"
}

Output

The output from a map state is a JSON array where each element is the output from the execution of the Iterator for a given item in the input array. The ResultPath can be used to define where in the JSON to insert the array of results. For example the below value will place the results array under a top-level JSON path called executionResults.

ResultPath: "$.executionResults"

ItemsPath

ItemsPath is very simple and is like InputPath except for the iterator you define in the map state. It tells the Iterator where the find the array of items to process in the input JSON object.

For example to access the items in the input JSON below you would use:

ItemsPath: "$.detail.items"

{
  "orderId": "12345678",
  "orderDate": "20190820101213",
  "detail": {
    "customerId": "1234",
    "deliveryAddress": "123, Seattle, WA",
    "deliverySpeed": "1-day",
    "paymentMethod": "aCreditCard",
    "items": [
      {
        "productName": "Agile Software Development",
        "category": "book",
        "price": 60.0,
        "quantity": 1
      },
      {
        "productName": "Domain-Driven Design",
        "category": "book",
        "price": 32.0,
        "quantity": 1
      },
      {
        "productName": "The Mythical Man Month",
        "category": "book",
        "price": 18.0,
        "quantity": 1
      },
      {
        "productName": "The Art of Computer Programming",
        "category": "book",
        "price": 180.0,
        "quantity": 1
      },
      {
        "productName": "Ground Coffee, Dark Roast",
        "category": "grocery",
        "price": 8.0,
        "quantity": 6
      }
    ]
  }
}

Combined With Nested Workflows

Now that AWS Step Functions supports both nested workflows and dynamic parallelism it is possible to apply another workflow you have built across many items in parallel, completely removing the need for the many hacks and workarounds need to achieve this previously.

For example imagine we had a workflow that can process a single book order with multiple steps such as taking payment, verifying stock etc. What happens when we want to process multiple book orders? Previously we would need to build a loop in the state machine using Choice states and tracking indexes and all manner of workarounds.

Now however we could simple define a second state machine that uses a Map state uses the first workflow as a nested workflow, allowing us to apply the first workflow to each individual book in the list of books ordered.

If you need a refresher on nested workflows in AWS Step Functions I previously wrote an article after their announcement, you can find it below.

  • AWS Step Functions Nested Workflows
    AWS Step Functions recently added nested workflows, a feature customers have been asking for a lot. In this article we take a look at the new features added for nested workflows.

References

Related Articles