AWS Batch

This guide covers deploying a Funnel server that leverages DynamoDB for storage and AWS Batch for task execution.

Setup

Get started by creating a compute environment, job queue and job definition using either the Funnel CLI or the AWS Batch web console. To manage the permissions of instanced AWS Batch jobs create a new IAM role. For the Funnel configuration outlined in this document, this role will need to provide read and write access to both S3 and DynamoDB.

Note: We recommend creating the Job Definition with Funnel by running: funnel aws batch create-job-definition. Funnel expects the JobDefinition to start a Funnel worker process with a specific configuration. Only advanced users should consider making any substantial changes to this Job Definition.

AWS Batch tasks, by default, launch the ECS Optimized AMI which includes an 8GB volume for the operating system and a 22GB volume for Docker image and metadata storage. The default Docker configuration allocates up to 10GB of this storage to each container instance. Read more about the default AMI. Due to these limitations, we recommend creating a custom AMI. Because AWS Batch has the same requirements for your AMI as Amazon ECS, use the default Amazon ECS-optimized Amazon Linux AMI as a base and change it to better suit your tasks.

Steps

For more information check out AWS Batch’s getting started guide.

Quickstart

$ funnel aws batch create-all-resources --region us-west-2

This command will create a compute environment, job queue, IAM role and job definition.

Configuring the Funnel Server

Below is an example configuration. Note that the Key and Secret fields are left blank in the configuration of the components. This is because Funnel will, by default, try to will try to automatically load credentials from the environment. Alternatively, you may explicitly set the credentials in the config.

Database: "dynamodb"
Compute: "aws-batch"
EventWriters:
  - "log"

Dynamodb:
  TableBasename: "funnel"
  Region: "us-west-2"
  Key: ""
  Secret: ""

Batch:
  JobDefinition: "funnel-job-def"
  JobQueue: "funnel-job-queue" 
  Region: "us-west-2"
  Key: ""
  Secret: ""
          
AmazonS3:
  Key: ""
  Secret: ""

Start the server

funnel server run --config /path/to/config.yaml

Known issues

The Task.Resources.DiskGb field does not have any effect. See issue 317.

Funnel