S3

Amazon S3

Funnel supports using AWS S3 for file storage.

The Amazon S3 storage client is enabled by default, and will try to automatically load credentials from the environment. Alternatively, you may explicitly set the credentials in the worker config:

AmazonS3:
  Disabled: false
  # The maximum number of times that a request will be retried for failures.
  MaxRetries: 10
  Key: ""
  Secret: ""

The Amazon S3 storage client also supports SSE-KMS and SSE-C configurations.

For SSE-KMS as long as your credentials can access the KMS key used for the given bucket, no special configuration is required. However, you can specifiy a specific KMS key if desired:

AmazonS3:
SSE:
    KMSKey: "1a03ce70-5f03-484e-8396-0e97de661b79"

For SSE-C:

Generate a key file:

openssl rand -out sse-c.key 32

Then configure the storage client to use it:

AmazonS3:
  SSE:
    CustomerKeyFile: "./sse-c.key"

Note that this file will need to be available to all Funnel workers.

Other S3 API Providers

Funnel also supports using non-Amazon S3 API providers (Ceph, Cleversafe, Minio, etc.) for file storage.

These other S3 storage clients are NOT enabled by default. You must configure them.

This storage client currently only supports the version 2 signing process.

GenericS3:
  - Disabled: false
    Endpoint: ""
    Key: ""
    Secret: ""

Example task

{
  "name": "Hello world",
  "inputs": [{
    "url": "s3://funnel-bucket/hello.txt",
    "path": "/inputs/hello.txt"
  }],
  "outputs": [{
    "url": "s3://funnel-bucket/output.txt",
    "path": "/outputs/hello-out.txt"
  }],
  "executors": [{
    "image": "alpine",
    "command": ["cat", "/inputs/hello.txt"],
    "stdout": "/outputs/hello-out.txt"
  }]
}