Cloud Storage Config

Captain supports indexing from both Amazon S3 and Google Cloud Storage buckets.

This guide walks through the process of connecting them with Captain.

Choose your cloud storage provider

AWS S3 Buckets
AWS S3 Buckets
GCP Storage Buckets
GCP Storage Buckets

AWS S3 Bucket Setup

Step 1: Create the Bucket

  1. Log in to the AWS Console

  2. Navigate to S3 → Buckets → Create bucket

    Captain recommends the following standard configuration:
    1. Select a region for the bucket
    2. Set a bucket name
    3. Keep General Purpose Storage selected if asked
    4. Keep ACLs disabled
    5. Keep Block all public access checked
    6. (Optional) Enable Bucket Versioning (for SOC 2 compliance)
    7. Once these settings are configured, the bucket is ready.

      Scroll down and click Create bucket

AWS S3 Access Keys

To index files from Amazon S3 buckets, you’ll need an AWS Access Key ID and Secret Access Key.

Step 1: Navigate to IAM Security Credentials

  1. Log in to the AWS Console
  2. Click on your account name in the top-right corner
  3. Select Security credentials from the dropdown menu
AWS Console - Security Credentials

Step 2: Create Access Key

  1. Scroll down to the Access keys section
  2. Click the Create access key button
AWS IAM - Create Access Key

Step 3: Retrieve Your Credentials

  1. Your Access Key ID and Secret Access Key will be displayed
  2. Important: This is the only time you can view the Secret Access Key
  3. Click Show to reveal the Secret Access Key
  4. Copy both the Access Key ID and Secret Access Key to a secure location
  5. Optionally, download the .csv file for safekeeping
AWS - Retrieve Access Key

Required IAM Permissions (AWS)

Your AWS access key needs the following permissions to work with Captain:

For read-only access to S3 buckets:

1{

2  "Version": "2012-10-17",

3  "Statement": [

4    {

5      "Effect": "Allow",

6      "Action": [

7        "s3:GetObject",

8        "s3:ListBucket"

9      ],

10      "Resource": [

11        "arn:aws:s3:::your-bucket-name",

12        "arn:aws:s3:::your-bucket-name/*"

13      ]

14    }

15  ]

16}

Replace your-bucket-name with your actual S3 bucket name.

Google Cloud Storage Bucket Setup

Step 1: Create the Bucket

  1. Go to Google Cloud Console
  2. Navigate to Cloud Storage → Buckets → Create
  3. Enter a unique bucket name (e.g., company-captain-documents)
  4. Select a Location type (Region, Dual-region, or Multi-region)
  5. Select Standard storage class for general use
  6. Under Access control, select “Uniform” (recommended)
  7. Click Create

Google Cloud Storage Credentials

To index files from Google Cloud Storage buckets, you’ll need a Service Account JSON key.

Step 1: Navigate to Service Accounts

  1. Go to Google Cloud Console
  2. Navigate to IAM & Admin → Service Accounts
  3. Click Create Service Account
Google Cloud - Service Accounts

Step 2: Create Service Account

  1. Enter a Service account name (e.g., captain-storage-access)
  2. Add a Service account description (optional but recommended)
  3. The Service account ID will be auto-generated
  4. Click Create and continue
Google Cloud - Create Service Account

Step 3: Grant Permissions

Under Grant this service account access to project, choose the appropriate role based on your needs:

For read-only access to buckets/objects:

  • Role: Storage Object Viewer (roles/storage.objectViewer)

For read/write access:

  • Role: Storage Object Admin (roles/storage.objectAdmin)

For full bucket management:

  • Role: Storage Admin (roles/storage.admin)

Click Continue → Done

Step 4: Create and Download JSON Key

  1. You’ll now see your new service account in the list
  2. Click on the service account name
  3. Navigate to the Keys tab
  4. Click Add Key → Create New Key
  5. Select JSON as the key type
  6. Click Create
Google Cloud - Create JSON Key

The JSON key file will automatically download to your computer. This file contains your service account credentials.

JSON Key File Format

Your downloaded JSON key will look like this:

1{

2  "type": "service_account",

3  "project_id": "your-project-id",

4  "private_key_id": "abc123...",

5  "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",

6  "client_email": "captain-storage-access@your-project.iam.gserviceaccount.com",

7  "client_id": "123456789",

8  "auth_uri": "https://accounts.google.com/o/oauth2/auth",

9  "token_uri": "https://oauth2.googleapis.com/token",

10  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",

11  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/..."

12}

Using Google Cloud Service Accounts with Captain

Store the JSON key file securely (e.g., in a secret management service or as an environment variable)

Troubleshooting

AWS Issues

Error: “Invalid AWS credentials”

  • Verify your Access Key ID and Secret Access Key are correct
  • Check that the access key is active in the IAM console
  • Ensure your IAM user/role has the necessary S3 permissions

Error: “Access Denied”

  • Verify your IAM permissions include s3:GetObject and s3:ListBucket
  • Check bucket policies and ensure they allow your IAM user/role
  • Verify the bucket region matches the bucket_region parameter

Google Cloud Issues

Error: “Invalid service account credentials”

  • Verify the JSON key file is valid and not corrupted
  • Check that the service account is enabled
  • Ensure the service account has the necessary Storage permissions

Error: “Permission denied”

  • Verify the service account has the appropriate Storage role
  • Check that the bucket exists and the service account has access
  • Review IAM permissions in the Google Cloud Console

Need Help?

If you encounter issues obtaining or using your cloud storage credentials, contact Captain support: