At a project I’ve been working on CloudFormation was used. It’s not my favorite method for Infrastructure as Code (IaC) and I wouldn’t choose it, however it’s sometimes not a decision you can make, but its definitely better than clickops. One drawback is that it is a bit limited in how to set up a dynamic set of resources when the number of resources are not known in advance.
There are times when it would be nice to be able perform a for loop in CloudFormation. At time of writing, this is not possible, however it does support the Transform parameter, where you can utilize lambda to perform changes.
Larger enterprises sometimes offer a set of pre-made infrastructure templates using the AWS Service Catalog, where this can come in handy.
Let’s say you for some reason want to set up a set of lambdas which is served through an API gateway that prints “Hello World!”, which takes api-gateway and a path as a simple example. And now you want to offer a way for whoever using this template to enter a list of strings where they can either set up arbitrary number of deployments, e.g. path1, path2 and path3. Normally this would mean three stacks and a lot of clicks.
A single template can be set up using:
AWSTemplateFormatVersion: "2010-09-09"
Description: AWS API Gateway with a Lambda Integration
Parameters:
PathName:
Type: String
ApiGatewayRestApi:
Type: String
Resources:
ApiGatewayMethod:
Type: AWS::ApiGatewayV2::Integration
Properties:
ApiId: !Ref ApiGatewayRestApi
Description: Test Integration
ConnectionType: INTERNET
CredentialsArn: !GetAtt ApiGatewayIamRole.Arn
PassthroughBehavior: WHEN_NO_MATCH
TimeoutInMillis: 29000
IntegrationMethod: POST
IntegrationType: AWS_PROXY
PayloadFormatVersion: "2.0"
IntegrationUri: !Sub "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${LambdaFunction.Arn}/invocations"
ApiGatewayIamRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Sid: ""
Effect: "Allow"
Principal:
Service:
- "apigateway.amazonaws.com"
Action:
- "sts:AssumeRole"
Path: "/"
Policies:
- PolicyName: LambdaAccess
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Action: "lambda:*"
Resource: !GetAtt LambdaFunction.Arn
ApiGatewayStage:
Type: AWS::ApiGatewayV2::Stage
Properties:
DeploymentId: !Ref ApiGatewayDeployment
Description: Lambda API Stage v0
ApiId: !Ref ApiGatewayRestApi
StageName: !Ref PathName
ApiGatewayDeployment:
Type: AWS::ApiGatewayV2::Deployment
DependsOn: ApiGatewayMethod
Properties:
Description: Lambda API Deployment
ApiId: !Ref ApiGatewayRestApi
LambdaFunction:
Type: AWS::Lambda::Function
Properties:
Code:
ZipFile: |
def handler(event, context):
response = {
'isBase64Encoded': False,
'statusCode': 200,
'headers': {},
'multiValueHeaders': {},
'body': 'Hello, World!'
}
return response
Description: AWS Lambda function
FunctionName: !Sub "lambda-function-${PathName}"
Handler: index.handler
MemorySize: 256
Role: !GetAtt LambdaIamRole.Arn
Runtime: python3.7
Timeout: 60
LambdaIamRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Principal:
Service:
- "lambda.amazonaws.com"
Action:
- "sts:AssumeRole"
Path: "/"
ApiGatewayResource:
Type: AWS::ApiGatewayV2::Route
DependsOn:
- LambdaFunction
- ApiGatewayMethod
Properties:
ApiId: !Ref ApiGatewayRestApi
RouteKey: !Sub "POST /lambda-function-${PathName}"
Target: !Join
- /
- - integrations
- !Ref ApiGatewayMethod
Outputs:
LambdaArn:
Value: !Ref LambdaFunction
Think of this as a module in Terraform. In order to deploy this multiple times in the same go we need to implement the Transform lambda function. Here is a python function that takes a commaseparated list and duplicates the template (with support for placeholders).
import copy
import json
import re
MACRO_NAME = 'ListForEach'
def process_template(template: dict, templateParameterValues: dict) -> tuple:
new_template = copy.deepcopy(template)
if not transform_template_section(template, new_template, templateParameterValues, 'Resources'):
return 'failed', template
if not transform_template_section(template, new_template, templateParameterValues, 'Outputs'):
return 'failed', template
return 'success', new_template
def transform_template_section(template: dict, new_template: dict, templateParameterValues: dict, section: str) -> bool:
for name, resource in template[section].items():
if MACRO_NAME in resource:
# Get the number of times to multiply the resource
commaSeparatedList = new_template[section][name].pop(MACRO_NAME)
# Remove the original resource from the template but take a local copy of it
resourceToMultiply = new_template[section].pop(name)
# Create a new block of the resource multiplied with names ending in the iterator and the placeholders substituted
resourcesAfterMultiplication = multiply(name, resourceToMultiply, commaSeparatedList, templateParameterValues)
if not set(resourcesAfterMultiplication.keys()) & set(new_template[section].keys()):
new_template[section].update(resourcesAfterMultiplication)
else:
return False
return True
def update_placeholder(resource_structure, iteration, listValues):
resourceString = json.dumps(resource_structure)
indexPlaceHolderCount = resourceString.count('%d') + resourceString.count('%s')
if indexPlaceHolderCount == 0:
print("No occurences of placeholder found in JSON, therefore nothing will be replaced")
return resource_structure
# Placeholders exists
print("Found {} occurrences of placeholder in JSON, replacing with iterator value {}".format(indexPlaceHolderCount, iteration))
regex = r"(\%d|\%s)"
matches = re.findall(regex, resourceString, re.MULTILINE)
placeHolderReplacementValues = [iteration if m == '%d' else listValues[iteration] for m in matches]
#Replace the decimal placeholders using the list - the syntax below expands the list
resourceString = resourceString % (*placeHolderReplacementValues,)
return json.loads(resourceString)
def multiply(resource_name, resource_structure, listValues, templateParameterValues):
resources = {}
#Loop according to the number of times we want to multiply, creating a new resource each time
if isinstance(listValues, dict) and 'Ref' in listValues:
listValues = templateParameterValues[listValues['Ref']]
count = len(listValues)
else:
count = listValues
for iteration in range(count):
print("Multiplying '{}', iteration count {}".format(resource_name,iteration))
multipliedResourceStructure = update_placeholder(resource_structure, iteration, listValues)
resources[resource_name+str(iteration)] = multipliedResourceStructure
return resources
def handler(event, context):
result = process_template(event['fragment'], event['templateParameterValues'])
return {
'requestId': event['requestId'],
'status': result[0],
'fragment': result[1],
}
Simply deploy it with:
---
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
ListForEach macro
To create resources from list of strings
Resources:
Macro:
Type: AWS::CloudFormation::Macro
Properties:
Name: ListForEach
FunctionName: !GetAtt ListForEachMacroFunction.Arn
ListForEachMacroFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: src
Handler: index.handler
Runtime: python3.9
Timeout: 5
Now everthing is set up, and you can start using it 🎉
Simply invoke it with the Transform parameter:
AWSTemplateFormatVersion: "2010-09-09"
Transform: ListForEach
Parameters:
PrivateSubdomains:
Type: CommaDelimitedList
Description: Comma separated list of private subdomains
Default: "priv1,priv2"
Resources:
PathToPublicBucket:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: https://some-bucket/cloudformation-manifest.yaml
Parameters:
PathName: "public-path-%d-%s"
ApiGatewayRestApi: !Ref ApiGatewayRestApi
ListForEach: !Ref PrivateSubdomains
Outputs:
PathToPrivateBucketArn:
Value: !Ref PathToPrivateBucket%d
ListForEach: !Ref PrivateSubdomains
Which will expand it to the following before running it
AWSTemplateFormatVersion: "2010-09-09"
Parameters:
PrivateSubdomains:
Type: CommaDelimitedList
Description: Comma separated list of private subdomains
Default: "priv1,priv2"
Resources:
PathToPublicBucket0:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: https://some-bucket/cloudformation-manifest.yaml
Parameters:
PathName: "public-path-0-priv1"
ApiGatewayRestApi: !Ref ApiGatewayRestApi
PathToPublicBucket1:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: https://some-bucket/cloudformation-manifest.yaml
Parameters:
PathName: "public-path-1-priv2"
ApiGatewayRestApi: !Ref ApiGatewayRestApi
Outputs:
PathToPrivateBucketArn0:
Value: !Ref PathToPrivateBucket0
PathToPrivateBucketArn1:
Value: !Ref PathToPrivateBucket1
You can see the full example in this repository