Indexing

Cloud Storage Config

Captain supports indexing from Amazon S3, Google Cloud Storage, Azure Blob Storage, Cloudflare R2, Supabase, Backblaze B2, and Dropbox.

This guide walks through the process of connecting them with Captain.

Choose your cloud storage provider

AWS S3 Buckets
AWS S3 Buckets
GCP Storage Buckets
GCP Storage Buckets
Azure Blob Storage
Azure Blob Storage
Cloudflare R2
Cloudflare R2
Supabase
Supabase
Backblaze B2
Backblaze B2
Dropbox
Dropbox
Google Drive
Google Drive

AWS S3 Bucket Setup

Step 1: Create the Bucket

  1. Log in to the AWS Console

  2. Navigate to S3 → Buckets → Create bucket

    Captain recommends the following standard configuration:
    1. Select a region for the bucket
    2. Set a bucket name
    3. Keep General Purpose Storage selected if asked
    4. Keep ACLs disabled
    5. Keep Block all public access checked
    6. (Optional) Enable Bucket Versioning (for SOC 2 compliance)
    7. Once these settings are configured, the bucket is ready.

      Scroll down and click Create bucket

AWS S3 Access Keys

To index files from Amazon S3 buckets, you’ll need an AWS Access Key ID and Secret Access Key.

Step 1: Navigate to IAM Security Credentials

  1. Log in to the AWS Console
  2. Click on your account name in the top-right corner
  3. Select Security credentials from the dropdown menu
AWS Console - Security Credentials

Step 2: Create Access Key

  1. Scroll down to the Access keys section
  2. Click the Create access key button
AWS IAM - Create Access Key

Step 3: Retrieve Your Credentials

  1. Your Access Key ID and Secret Access Key will be displayed
  2. Important: This is the only time you can view the Secret Access Key
  3. Click Show to reveal the Secret Access Key
  4. Copy both the Access Key ID and Secret Access Key to a secure location
  5. Optionally, download the .csv file for safekeeping
AWS - Retrieve Access Key

Required IAM Permissions (AWS)

Your AWS access key needs the following permissions to work with Captain:

For read-only access to S3 buckets:

1{
2  "Version": "2012-10-17",
3  "Statement": [
4    {
5      "Effect": "Allow",
6      "Action": [
7        "s3:GetObject",
8        "s3:ListBucket"
9      ],
10      "Resource": [
11        "arn:aws:s3:::your-bucket-name",
12        "arn:aws:s3:::your-bucket-name/*"
13      ]
14    }
15  ]
16}

Replace your-bucket-name with your actual S3 bucket name.

Google Cloud Storage Bucket Setup

Step 1: Create the Bucket

  1. Go to Google Cloud Console
  2. Navigate to Cloud Storage → Buckets → Create
  3. Enter a unique bucket name (e.g., company-captain-documents)
  4. Select a Location type (Region, Dual-region, or Multi-region)
  5. Select Standard storage class for general use
  6. Under Access control, select “Uniform” (recommended)
  7. Click Create

Google Cloud Storage Credentials

To index files from Google Cloud Storage buckets, you’ll need a Service Account JSON key.

Step 1: Navigate to Service Accounts

  1. Go to Google Cloud Console
  2. Navigate to IAM & Admin → Service Accounts
  3. Click Create Service Account
Google Cloud - Service Accounts

Step 2: Create Service Account

  1. Enter a Service account name (e.g., captain-storage-access)
  2. Add a Service account description (optional but recommended)
  3. The Service account ID will be auto-generated
  4. Click Create and continue
Google Cloud - Create Service Account

Step 3: Grant Permissions

Under Grant this service account access to project, choose the appropriate role based on your needs:

For read-only access to buckets/objects:

  • Role: Storage Object Viewer (roles/storage.objectViewer)

For read/write access:

  • Role: Storage Object Admin (roles/storage.objectAdmin)

For full bucket management:

  • Role: Storage Admin (roles/storage.admin)

Click Continue → Done

Step 4: Create and Download JSON Key

  1. You’ll now see your new service account in the list
  2. Click on the service account name
  3. Navigate to the Keys tab
  4. Click Add Key → Create New Key
  5. Select JSON as the key type
  6. Click Create
Google Cloud - Create JSON Key

The JSON key file will automatically download to your computer. This file contains your service account credentials.

JSON Key File Format

Your downloaded JSON key will look like this:

1{
2  "type": "service_account",
3  "project_id": "your-project-id",
4  "private_key_id": "abc123...",
5  "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
6  "client_email": "captain-storage-access@your-project.iam.gserviceaccount.com",
7  "client_id": "123456789",
8  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
9  "token_uri": "https://oauth2.googleapis.com/token",
10  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
11  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/..."
12}

Using Google Cloud Service Accounts with Captain

Store the JSON key file securely (e.g., in a secret management service or as an environment variable)

Azure Blob Storage Setup

Step 1: Create a Storage Account

  1. Log in to the Azure Portal
  2. Navigate to Storage accounts → Create
  3. Select your Subscription and Resource group (or create a new one)
  4. Enter a Storage account name (e.g., captaindocuments)
  5. Select a Region closest to your operations
  6. Select Standard performance and LRS (Locally-redundant storage) for general use
  7. Click Review + create → Create

Step 2: Create a Container

  1. Open your new storage account
  2. Navigate to Data storage → Containers
  3. Click + Container
  4. Enter a container name (e.g., documents)
  5. Set Private access level (no anonymous access)
  6. Click Create

Azure Blob Storage Credentials

To index files from Azure Blob Storage, you’ll need your Storage Account Name and Account Key.

Step 1: Get Your Account Name and Key

  1. Open your storage account in the Azure Portal
  2. Navigate to Security + networking → Access keys
  3. Your Storage account name is displayed at the top
  4. Click Show next to either key to reveal the Account Key (base64-encoded)
  5. Copy both values

Important: Treat the account key like a password. Store it securely (e.g., in a secret management service or as an environment variable).

Step 2: Use with Captain

Pass the account name, account key, and container name when calling the indexing endpoint:

$curl -X POST https://api.runcaptain.com/v2/collections/my_collection/index/azure \
>     -H "Authorization: Bearer $CAPTAIN_API_KEY" \
>     -H "Content-Type: application/json" \
>     -d '{
>       "container_name": "documents",
>       "account_name": "captaindocuments",
>       "account_key": "your_account_key_base64",
>       "processing_type": "advanced"
>     }'

Required Azure Permissions

The account key provides full access to all containers in the storage account. For more granular control, you can use Shared Access Signatures (SAS) with the following minimum permissions:

  • Read. to access blob contents
  • List. to enumerate blobs in the container

You can generate a SAS token from the Azure Portal under your storage account’s Shared access signature settings.

Cloudflare R2 Setup

Step 1: Create an R2 Bucket

  1. Log in to the Cloudflare Dashboard
  2. Navigate to R2 Object Storage → Create bucket
  3. Enter a bucket name (e.g., captain-documents)
  4. Select a location hint (optional. R2 automatically distributes globally)
  5. Click Create bucket

Cloudflare R2 Credentials

To index files from Cloudflare R2, you’ll need your Account ID, an Access Key ID, and a Secret Access Key.

Step 1: Find Your Account ID

  1. Log in to the Cloudflare Dashboard
  2. Your Account ID is visible in the URL: https://dash.cloudflare.com/<account_id>
  3. You can also find it on the R2 Overview page in the right sidebar

Step 2: Create an R2 API Token

  1. Navigate to R2 Object Storage → Manage R2 API Tokens
  2. Click Create API token
  3. Enter a token name (e.g., captain-read-access)
  4. Under Permissions, select Object Read only
  5. Under Specify bucket(s), select your bucket or allow access to all buckets
  6. Click Create API Token

Step 3: Retrieve Your Credentials

  1. Your Access Key ID and Secret Access Key will be displayed
  2. Important: This is the only time you can view the Secret Access Key
  3. Copy both values to a secure location

Step 4: Use with Captain

Pass the account ID, access key ID, secret access key, and bucket name when calling the indexing endpoint:

$curl -X POST https://api.runcaptain.com/v2/collections/my_collection/index/r2 \
>     -H "Authorization: Bearer $CAPTAIN_API_KEY" \
>     -H "Content-Type: application/json" \
>     -d '{
>       "bucket_name": "captain-documents",
>       "account_id": "your_cloudflare_account_id",
>       "access_key_id": "your_r2_access_key_id",
>       "secret_access_key": "your_r2_secret_access_key",
>       "processing_type": "advanced"
>     }'

R2 Jurisdictions

R2 supports jurisdiction-restricted storage. You can optionally specify a jurisdiction parameter:

  • default. Global (no restriction). This is the default.
  • eu. EU-only data residency
  • fedramp. FedRAMP-compliant storage

Required R2 Permissions

Your R2 API token needs the following minimum permissions:

  • Object Read. to access object contents
  • List. to enumerate objects in the bucket

For more granular control, create a token scoped to a specific bucket rather than all buckets.

Supabase Setup

Supabase Storage is S3-compatible, so Captain connects to it through its S3 endpoint.

Step 1: Create a Storage Bucket

  1. Log in to the Supabase Dashboard and open your project
  2. Navigate to Storage → New bucket
  3. Enter a bucket name (e.g., documents)
  4. Choose whether the bucket is public or private (private is fine. Captain authenticates with S3 credentials)
  5. Click Create bucket

Supabase Credentials

To index files from Supabase, you’ll need the S3 endpoint URL, the region, an Access Key ID, and a Secret Access Key.

Step 1: Enable the S3 Connection

  1. In the Supabase Dashboard, navigate to Storage → S3 Connection
  2. Enable the S3 connection if it isn’t already enabled
  3. Copy the Endpoint (e.g., https://abcdefgh.storage.supabase.co/storage/v1/s3) and the Region shown on this page

Pass this endpoint as endpoint_url. The subdomain is your project ref. for https://abcdefgh.storage.supabase.co/storage/v1/s3 the project ref is abcdefgh, but Captain only needs the full endpoint_url copied from Storage → S3 Connection.

Step 2: Create S3 Access Keys

  1. On the S3 Connection page, click Create new access key (under S3 Access Keys)
  2. Enter a description (e.g., captain-read-access)
  3. Click Create access key

Step 3: Retrieve Your Credentials

  1. Your Access Key ID and Secret Access Key will be displayed
  2. Important: This is the only time you can view the Secret Access Key
  3. Copy both values to a secure location

Step 4: Use with Captain

Pass the endpoint URL, bucket name, access key ID, and secret access key when calling the indexing endpoint. region is optional and defaults to us-east-1:

$curl -X POST https://api.runcaptain.com/v2/collections/my_collection/index/supabase \
>     -H "Authorization: Bearer $CAPTAIN_API_KEY" \
>     -H "Content-Type: application/json" \
>     -d '{
>       "bucket_name": "documents",
>       "endpoint_url": "https://abcdefgh.storage.supabase.co/storage/v1/s3",
>       "access_key_id": "your_supabase_access_key",
>       "secret_access_key": "your_supabase_secret_key",
>       "region": "us-east-1",
>       "processing_type": "advanced"
>     }'

Endpoint URL

Captain connects using the endpoint_url you pass:

  • endpoint_url. The full S3-compatible endpoint URL, copied from Storage → S3 Connection. It has the form https://<project_ref>.storage.supabase.co/storage/v1/s3, where <project_ref> is your project’s subdomain (e.g. abcdefgh).

Required Supabase Permissions

The S3 access key needs read and list access to the bucket you want to index. Access keys created from Storage → S3 Connection carry the necessary permissions by default.

Backblaze B2 Setup

Backblaze B2 is S3-compatible, so Captain connects to it through its S3 endpoint.

Step 1: Create a Bucket

  1. Log in to the Backblaze B2 dashboard and open Buckets
  2. Click Create a Bucket
  3. Enter a bucket name (e.g., documents)
  4. Choose whether the bucket is public or private (private is fine. Captain authenticates with S3 credentials)
  5. Click Create a Bucket
  6. Open the bucket’s details page. it shows the bucket’s Endpoint (in the form https://s3.<region>.backblazeb2.com, e.g. https://s3.us-west-004.backblazeb2.com) and its Region

Backblaze B2 Credentials

To index files from Backblaze B2, you’ll need the S3 endpoint URL, the region, an Access Key ID (Application Key ID), and a Secret Access Key (Application Key).

Step 1: Find the Endpoint and Region

  1. In the Backblaze B2 dashboard, open Buckets and select your bucket
  2. On the bucket’s details page, copy the Endpoint (e.g., https://s3.us-west-004.backblazeb2.com) and the Region (e.g., us-west-004) shown there

Pass this endpoint as endpoint_url. The region in the endpoint (e.g. us-west-004) must match the region you send.

Step 2: Create an Application Key

  1. Navigate to App Keys in the Backblaze B2 dashboard
  2. Click Add a New Application Key (do not use the master application key. it cannot be used with the S3-compatible API)
  3. Give the key a name and, optionally, scope it to a specific bucket with read access
  4. Click Create New Key

Step 3: Retrieve Your Credentials

  1. Backblaze displays a keyID and an applicationKey
  2. Important: the applicationKey is shown only once, at creation time
  3. Copy both values to a secure location. use the keyID as access_key_id and the applicationKey as secret_access_key

Step 4: Use with Captain

Pass the endpoint URL, bucket name, access key ID (Application Key ID), secret access key (Application Key), and region when calling the indexing endpoint. Backblaze B2’s S3 API requires v4 signatures over HTTPS, which Captain uses by default:

$curl -X POST https://api.runcaptain.com/v2/collections/my_collection/index/backblaze \
>     -H "Authorization: Bearer $CAPTAIN_API_KEY" \
>     -H "Content-Type: application/json" \
>     -d '{
>       "bucket_name": "documents",
>       "endpoint_url": "https://s3.us-west-004.backblazeb2.com",
>       "access_key_id": "your_b2_application_key_id",
>       "secret_access_key": "your_b2_application_key",
>       "region": "us-west-004",
>       "processing_type": "advanced"
>     }'

Endpoint URL

Captain connects using the endpoint_url you pass:

  • endpoint_url. The full S3-compatible endpoint URL, copied from the bucket’s details page. It has the form https://s3.<region>.backblazeb2.com, where <region> is your bucket’s region (e.g. us-west-004).

Required Backblaze B2 Permissions

The Application Key needs read (and list) access to the bucket you want to index. Create the key on the App Keys page. the master application key cannot be used with the S3-compatible API.

Dropbox Setup

Step 1: Create a Dropbox App

  1. Log in to the Dropbox App Console
  2. Click Create app
  3. Under Choose an API, select Scoped access
  4. Under Choose the type of access you need, select Full Dropbox (or App folder to scope to a single folder)
  5. Enter an app name (e.g., captain-indexing)
  6. Click Create app

Dropbox Credentials

To index files from Dropbox, you’ll need a Dropbox access token.

Step 1: Grant the Required Scopes

  1. Open your app in the Dropbox App Console
  2. Navigate to the Permissions tab
  3. Enable the following scopes:
    • files.metadata.read. to list files and read their metadata
    • files.content.read. to download file contents for indexing
    • sharing.write (optional). lets Captain store a permanent Dropbox link as each document’s source.uri, so results link straight back to the original file. Without it, indexing still works and the source.uri falls back to the file’s Dropbox path.
  4. Click Submit to save the permission changes

Step 2: Generate an Access Token

  1. Navigate to the Settings tab
  2. Under OAuth 2, find Generated access token
  3. Click Generate
  4. Copy the token to a secure location

Tokens generated from the App Console are scoped to your own Dropbox account and are short-lived. This is ideal for one-shot indexing jobs, which complete in minutes. Regenerate a token whenever you start a new job.

Step 3: Use with Captain

Pass the access token when calling the indexing endpoint. The /index/dropbox endpoint indexes the entire account recursively. To scope a job to a single folder or file, use /index/dropbox/directory (with directory_path) or /index/dropbox/file (with file_path) instead:

$curl -X POST https://api.runcaptain.com/v2/collections/my_collection/index/dropbox \
>     -H "Authorization: Bearer $CAPTAIN_API_KEY" \
>     -H "Content-Type: application/json" \
>     -d '{
>       "dropbox_access_token": "your_dropbox_access_token",
>       "processing_type": "advanced"
>     }'

Required Dropbox Scopes

Your Dropbox access token needs the following minimum scopes:

  • files.metadata.read. to list files and read their metadata
  • files.content.read. to download object contents for indexing

Optionally, add sharing.write so Captain can store a permanent, clickable Dropbox link as each indexed document’s source.uri. Without it, indexing works normally and source.uri falls back to the file’s Dropbox path.

Enable these on the Permissions tab of the App Console, then regenerate the access token so the new scopes take effect.

Google Drive (headless) Setup

Unlike the other providers, Google Drive uses service-account domain-wide delegation — no per-user OAuth pop-up. Your Google Workspace admin authorizes one service account once, and Captain reads Drive as a specific user you name on each request: a single file, a folder (recursively), or that user’s entire Drive. Use this to index Drive content at scale, on a schedule, or on behalf of many users, with no login step in the loop.

The setup below is done once by someone with Google Workspace super administrator rights on your domain (about 15 minutes). After that, indexing is a plain API call.

Step 1: Create a service account and key

You need a Google Cloud project to hold the service account. If your organization doesn’t use Google Cloud, that’s fine — any Google account can create a project for free, and this use case requires no billing.

  1. Go to console.cloud.google.com.
  2. Create a project (or pick an existing one): open the project picker in the top bar, click New Project, name it (for example captain-drive-indexing), and click Create.
  3. Enable the Drive API: go to APIs & Services → Enable APIs and services, search for “Google Drive API”, and click Enable.
  4. Create the service account: go to IAM & Admin → Service Accounts, click Create service account, give it a name (for example captain-indexing), and click Create and continue. Skip the optional roles and click Done. No project roles are needed; this account’s access comes entirely from the Workspace delegation grant in Step 2.
  5. Create a key: open the service account, go to the Keys tab, and click Add key → Create new key → JSON → Create. A .json file downloads. This is the service_account_json you’ll send to Captain — keep it secret.
  6. Note the service account’s numeric Client ID (also called the “Unique ID” or “OAuth2 Client ID”) from the Details tab. You’ll need it in Step 2.

Google does not offer a way to create a service account entirely inside the Admin console — the account itself must be created in Google Cloud, as above. Only the delegation grant (Step 2) lives in the Admin console.

Step 2: Authorize the service account (domain-wide delegation)

This is the grant that lets Captain read Drive as your users. It’s done in the Admin console (admin.google.com), a different console from Step 1.

  1. Go to admin.google.com, then Security → Access and data control → API controls.
  2. In the Domain-wide delegation panel, click Manage Domain-Wide Delegation.
  3. Click Add new and enter:
    • Client ID: the numeric Client ID from Step 1 (not the service account’s email address).
    • OAuth scopes: exactly https://www.googleapis.com/auth/drive.readonly
  4. Click Authorize.

Grant only drive.readonly — Captain never writes to or deletes from Drive. The grant can take a few minutes to propagate (occasionally up to ~30); if your first call returns unauthorized_client, wait and retry.

Google Drive Credentials

Every request sends the service-account key plus a subject_email — the Workspace user whose Drive Captain reads. Only what that user can see is indexed.

Choosing subject_email:

  • A specific person’s Drive — their email (e.g. analyst@yourco.com).
  • A Shared Drive / team content — any user who is a member of that Shared Drive. Files that user can’t see are skipped.
  • An org-wide identity — many teams create a dedicated Workspace user (e.g. indexing@yourco.com), share the relevant Drives with it, and always impersonate that account, keeping the indexed scope explicit and auditable.

Index a single file, a folder (recursively), or the user’s entire Drive:

curl
$# One folder, recursively
$curl -X POST "https://api.runcaptain.com/v2/collections/{collection_name}/index/gdrive/directory" \
>  -H "Authorization: Bearer $CAPTAIN_API_KEY" \
>  -H "Content-Type: application/json" \
>  -d '{
>    "service_account_json": "<full contents of the JSON key file, as a string>",
>    "subject_email": "analyst@yourco.com",
>    "folder_id": "1MHp...",
>    "processing_type": "advanced"
>  }'

Swap the endpoint and id field for other scopes:

  • Single file — .../index/gdrive/file with "file_id": "1AbC..." (the /d/<id>/ part of a file URL).
  • Entire Drive — .../index/gdrive with no id field.

folder_id is the trailing segment of a folder URL (drive.google.com/drive/folders/<folder_id>).

What gets indexed: regular files (PDF, DOCX, XLSX, images, etc.) normally; native Google Docs and Slides export to PDF and Sheets export to XLSX automatically; Shared Drives are included. Google Forms/Sites are skipped, as is any file the subject_email user can’t access (logged and skipped — the job continues).

Troubleshooting

AWS Issues

Error: “Invalid AWS credentials”

  • Verify your Access Key ID and Secret Access Key are correct
  • Check that the access key is active in the IAM console
  • Ensure your IAM user/role has the necessary S3 permissions

Error: “Access Denied”

  • Verify your IAM permissions include s3:GetObject and s3:ListBucket
  • Check bucket policies and ensure they allow your IAM user/role
  • Verify the bucket region matches the bucket_region parameter

Google Cloud Issues

Error: “Invalid service account credentials”

  • Verify the JSON key file is valid and not corrupted
  • Check that the service account is enabled
  • Ensure the service account has the necessary Storage permissions

Error: “Permission denied”

  • Verify the service account has the appropriate Storage role
  • Check that the bucket exists and the service account has access
  • Review IAM permissions in the Google Cloud Console

Azure Issues

Error: “Invalid credentials”

  • Verify the account name matches your storage account exactly
  • Check that the account key is the full base64-encoded key (not truncated)
  • Ensure the storage account exists and is active

Error: “Container not found”

  • Verify the container name matches exactly (case-sensitive)
  • Check that the container exists in the storage account
  • Ensure the storage account is in the correct subscription

Cloudflare R2 Issues

Error: “Invalid credentials”

  • Verify your Account ID matches the one in your Cloudflare dashboard URL
  • Check that the Access Key ID and Secret Access Key are from an R2 API token (not a general Cloudflare API token)
  • Ensure the API token has not been revoked

Error: “Access Denied”

  • Verify the R2 API token has Object Read permissions
  • Check that the token is scoped to the correct bucket (or all buckets)
  • Ensure the bucket exists and is in the correct jurisdiction

Supabase Issues

Error: “Invalid credentials”

  • Verify your endpoint_url points at https://<project_ref>.storage.supabase.co/storage/v1/s3 (copied from Storage → S3 Connection)
  • Check that the Access Key ID and Secret Access Key are S3 access keys from Storage → S3 Connection (not the project’s anon/service_role API keys)
  • Ensure the S3 connection is enabled for your project

Error: “Access Denied”

  • Confirm the S3 access key has read and list access to the target bucket
  • Check that the region matches the one shown on the S3 Connection page (defaults to us-east-1)
  • Ensure the bucket exists and the name is spelled correctly

Backblaze B2 Issues

Error: “Invalid credentials”

  • Verify you are using an Application Key (created on the App Keys page), not the master application key. the master key cannot be used with the S3-compatible API
  • Check that the keyID is passed as access_key_id and the applicationKey as secret_access_key
  • Ensure the Application Key has not been deleted

Error: “Access Denied”

  • Confirm the Application Key has read (and list) access to the target bucket
  • Check that the endpoint_url and region match the values shown on the bucket’s details page (the region in https://s3.<region>.backblazeb2.com must match region)
  • Ensure the bucket exists and the name is spelled correctly

Dropbox Issues

Error: “Invalid credentials”

  • Verify the access token is correct and copied in full (not truncated)
  • Tokens generated from the App Console are short-lived. Regenerate a fresh token if the job was created a while after the token was minted
  • Ensure the token belongs to the Dropbox account that holds the files you want to index

Error: “Access Denied”

  • Verify the app has the files.metadata.read and files.content.read scopes enabled on the Permissions tab
  • Regenerate the access token after changing scopes. tokens minted before a scope change do not carry the new permissions
  • Check that the path exists and is spelled correctly (Dropbox paths are case-insensitive but must otherwise match)

Need Help?

If you encounter issues obtaining or using your cloud storage credentials, contact Captain support:

© 2026 Captain