Set Up Sync

Set up Sync with your cloud storage

Pick a provider to jump to its setup. The same sync behavior (cadence, change detection, deletion propagation, and real-time events) applies to every provider.

AWS S3

Cloudflare R2

Supabase

Backblaze B2

How Sync works

One setup, then hands-off. Point a sync at a bucket. Captain indexes the existing files, then tracks the source so your search results always reflect what is in storage.

Many sources, one collection. Set up a sync for each bucket you want included, and Captain keeps all of them current in the same collection. Each sync tracks its own source independently, so a change or deletion in one source only affects the files that came from it.

Two mechanisms keep a collection current:

• Scheduled reconciliation runs on the cadence you choose. Captain lists the source, compares it against what it has already indexed, and applies the difference: new files get indexed, changed files get re-indexed, and removed files follow your deletion policy.
• Real-time events are an optional add-on. Forward change notifications from your source and updates land within seconds. Delivery is best-effort, so reconciliation always stays on as the backstop.

AWS S3

Authentication

An AWS S3 sync can authenticate two ways.

Role assumption (recommended). Create an IAM role in your own account that grants read access to the bucket, and Captain reads through that role using a Captain-issued external ID. No long-lived keys leave your account. The setup is the same as S3 indexing, so follow the S3 Cross-Account IAM guide, then pass an auth block of type assume_role with your role_arn and external_id.

Access key. Pass an auth block of type access_key with an access_key_id and secret_access_key. Captain stores the secret securely and never returns it. Use this when a role is not an option. It is also how the S3-compatible stores below authenticate, since they do not support IAM roles. For step-by-step credential setup with screenshots, see Connect Cloud Storage.

Create a sync

Creating a sync starts the first full index of the collection and returns the sync record right away. The indexing itself runs in the background.

1
import requests
2
3
BASE_URL = "https://api.runcaptain.com"
4
API_KEY = "cap_your_api_key"
5
6
resp = requests.post(
7
    f"{BASE_URL}/v2/collections/company-knowledge/sync/s3",
8
    headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
9
    json={
10
        "bucket": "my-company-docs",
11
        "prefix": "knowledge-base/",
12
        "region": "us-east-1",
13
        "auth": {
14
            "type": "assume_role",
15
            "role_arn": "arn:aws:iam::123456789012:role/captain-s3-read",
16
            "external_id": "captain-abc123",
17
        },
18
        "processing_type": "advanced",
19
        "deletion_policy": "mirror",
20
        "sync_interval_minutes": 15,
21
    },
22
)
23
print(resp.json())

A few fields worth calling out: prefix scopes the sync to part of the bucket (an empty prefix syncs everything), include_patterns and exclude_patterns are glob filters over object keys, and processing_type picks the parsing tier (advanced or basic). deletion_policy and sync_interval_minutes are covered below.

Cadence

sync_interval_minutes sets how often scheduled reconciliation runs.

Set sync_interval_minutes to null (or leave it out) for a manual sync with no scheduled reconciliation. Real-time events and on-demand reconcile still work. When changes need to land faster than every five minutes, use real-time events rather than a shorter cadence. The cadence is a backstop interval, not a latency guarantee.

Change the cadence at any time:

1
requests.patch(
2
    f"{BASE_URL}/v2/syncs/{sync_id}",
3
    headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
4
    json={"sync_interval_minutes": 60},
5
)

Change detection

Reconciliation decides what changed by comparing each object’s ETag, a content hash, against the value Captain recorded when it last indexed the object. It does not compare file size, so an in-place overwrite that keeps the same byte count is still caught and re-indexed. Objects whose content has not changed are skipped, so each pass only does work for what actually changed.

Deletion propagation

When an object is no longer in the bucket, deletion_policy decides what happens to its indexed copy:

Removed objects are matched by their exact bucket and key, so deletion targets the right document.

Real-time events

For faster propagation, enroll the event webhook. Captain returns a subscribe URL and the steps for forwarding object create and remove notifications to it.

1
resp = requests.post(
2
    f"{BASE_URL}/v2/syncs/{sync_id}/webhooks",
3
    headers={"Authorization": f"Bearer {API_KEY}"},
4
)
5
print(resp.json()["subscribe_url"])
6
print(resp.json()["instructions"])

Events sit alongside scheduled reconciliation rather than replacing it. Delivery is at-least-once and can drop or reorder messages, so keep a cadence set even with events enabled.

Reconcile on demand

Run a reconciliation right away instead of waiting for the next scheduled tick. It returns the changes it found and applied.

1
resp = requests.post(
2
    f"{BASE_URL}/v2/syncs/{sync_id}/reconcile",
3
    headers={"Authorization": f"Bearer {API_KEY}"},
4
)
5
print(resp.json())
6
# {"sync_id": "...", "job_id": "...", "added": 0, "modified": 1,
7
#  "removed": 0, "unchanged": 0, "deleted_documents": 0}

Pause, resume, and delete

Pause a sync to stop syncing without losing its configuration: send a PATCH to /v2/syncs/{sync_id} with {"status": "paused"}, and resume with {"status": "active"}.

Delete a sync with DELETE /v2/syncs/{sync_id}. This is a soft delete: syncing stops, the record is marked inactive, and its per-object sync state is cleared. The history is kept.

More cloud storage providers

Sync also works with Cloudflare R2, Supabase Storage, and Backblaze B2. Each has its own create endpoint that takes the fields native to that store. These stores do not support IAM roles, so they authenticate with an access key. Once a sync exists it is managed with the same endpoints, whichever store it came from.

1
resp = requests.post(
2
    f"{BASE_URL}/v2/collections/company-knowledge/sync/r2",
3
    headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
4
    json={
5
        "bucket": "my-bucket",
6
        "prefix": "docs/",
7
        "account_id": "your-account-id",
8
        "access_key_id": "your-r2-key-id",
9
        "secret_access_key": "your-r2-secret",
10
        "jurisdiction": "default",  # "default", "eu", or "fedramp"
11
        "processing_type": "advanced",
12
        "sync_interval_minutes": 15,
13
    },
14
)

Cadence, deletion propagation, events, and on-demand reconcile all behave the same as they do for AWS S3.

How it fits together

A common production setup: create the sync with a cadence such as 15 and deletion_policy: mirror, enroll the event webhook for faster updates, and leave scheduled reconciliation on as the backstop. Reconciliation catches any events that were dropped and brings the collection back in line with the bucket.

Endpoint reference

Full request and response schemas are in the Sync section of the API Reference. At a glance: