Boilerplate for S3 Batch Operation Lambda

S3 Batch Operation provide a simple way to process a number of files stored in an S3 bucket with a Lambda function. However, the Lambda function must return particular Response Codes. Below is an example of a Lambda function written in Python that works with AWS S3 Batch Operations.


Parsing S3 Inventory CSV output in Python

S3 Inventory is a great way to access a large number of keys in an S3 Bucket. Its output is easily parsed by AWS Athena, enabling queries across the key names (e.g. find all keys ending with .png)

However, sometimes you just need to list all of the keys mentioned in the S3 Inventory output (e.g. populating an SQS queue with every keyname mentioned in an inventory output). The following code is an example of doing such task in Python:


A PIL-friendly class for S3 objects

Here's a quick example of creating an file-like object in Python that represents an object on S3 and plays nicely with PIL. This ended up being overkill for my needs but I figured somebody might get some use out of it.


Using CloudFormation's Fn::Sub with Bash parameter substitution

Let's say that you need to inject a large bash script into a CloudFormation AWS::EC2::Instance Resource's UserData property. CloudFormation makes this easy with the Fn::Base64 intrinsic function:

AWSTemplateFormatVersion: '2010-09-09'

Resources:
  VPNServerInstance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: ami-efd0428f
      InstanceType: m3.medium
      UserData:
        Fn::Base64: |
          #!/bin/sh
          echo "Hello world"

In your bash script, you may even want to reference a parameter created elsewhere in the CloudFormation template. This is no problem with Cloudformation's Fn::Sub instrinsic function:

AWSTemplateFormatVersion: '2010-09-09'

Parameters:
  Username:
    Description: Username
    Type: String
    MinLength: '1'
    MaxLength: '255'
    AllowedPattern: '[a-zA-Z][a-zA-Z0-9]*'
    ConstraintDescription: must begin with a letter and contain only alphanumeric
      characters.

Resources:
  VPNServerInstance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: ami-efd0428f
      InstanceType: m3.medium
      UserData:
        Fn::Base64: !Sub |
          #!/bin/sh
          echo "Hello ${Username}"

The downside of the Fn::Sub function is that it does not play nice with bash' parameter substitution expressions:

AWSTemplateFormatVersion: '2010-09-09'

Parameters:
  Username:
    Description: Username
    Type: String
    MinLength: '1'
    MaxLength: '255'
    AllowedPattern: '[a-zA-Z][a-zA-Z0-9]*'
    ConstraintDescription: must begin with a letter and contain only alphanumeric
      characters.

Resources:
  VPNServerInstance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: ami-efd0428f
      InstanceType: m3.medium
      UserData:
        Fn::Base64: !Sub |
          #!/bin/sh
          echo "Hello ${Username}"
          FOO=${FOO:-'bar'}

The above template fails validation:

$ aws cloudformation validate-template --template-body file://test.yaml

An error occurred (ValidationError) when calling the ValidateTemplate operation: Template error: variable names in Fn::Sub syntax must contain only alphanumeric characters, underscores, periods, and colons

The work-around is to rely on another intrinsic function: Fn::Join:

AWSTemplateFormatVersion: '2010-09-09'

Parameters:
  Username:
    Description: Username
    Type: String
    MinLength: '1'
    MaxLength: '255'
    AllowedPattern: '[a-zA-Z][a-zA-Z0-9]*'
    ConstraintDescription: must begin with a letter and contain only alphanumeric
      characters.

Resources:
  VPNServerInstance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: ami-efd0428f
      InstanceType: m3.medium
      UserData:
        Fn::Base64: !Join
          - '\n'
          - - !Sub |
              #!/bin/sh
              echo "Hello ${Username}"
            - |
              FOO=${FOO:-'bar'}

This allows you to mix CloudFormation substitutions with Bash parameter substititions.


Bonus

While we're talking about CloudFormation, another good trick comes from cloudonaut.io regarding using a Optional Parameter in CloudFormation.

Parameters:
  KeyName:
    Description: (Optional) Select an ssh key pair if you will need SSH access to the machine
    Type: String

Conditions:
  HasKeyName:
    Fn::Not:
    - Fn::Equals:
      - ''
      - Ref: KeyName

Resources:
  VPNServerInstance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: ami-efd0428f
      InstanceType: m3.medium
      KeyName:
        Fn::If:
          - HasKeyName
          - !Ref KeyName
          - !Ref AWS::NoValue

Note that the KeyName has Type: String. While Type: AWS::EC2::KeyPair::KeyName would likely be a better user experience as it would render a dropdown of all keys, it does not allow for empty values:

... if you use the AWS::EC2::KeyPair::KeyName parameter type, AWS CloudFormation validates the input value against users' existing key pair names before it creates any resources, such as Amazon EC2 instances.