Multimodal Search

Captain supports native multimodal search across text documents, images, video, and audio files. Upload any combination of file types to a single collection, and search across all of them with one query. Captain handles format detection, media segmentation, embedding, and cross-modal ranking automatically.

Benchmark Results

On MRAG-Bench (ICLR 2025), a standardized academic benchmark for vision-centric retrieval with 16,130 images and 1,251 questions, Captain achieves 81.3% retrieval accuracy-outperforming every end-to-end RAG system tested in the paper, including GPT-4o with CLIP retrieval (68.96%). Full results, methodology, and evaluation code are available in our open-source evaluation repository.

Supported File Types

How It Works

Captain uses a dual embedding strategy for media files. Each image, video segment, or audio segment is embedded in two ways:

1. Native multimodal embedding (3072 dimensions): The raw bytes-image pixels, audio waveform, video frames-are embedded directly. This captures the actual visual, auditory, or temporal content.

2. Text embedding (1024 dimensions): A model generates a structured description of what the media actually contains (a transcript for audio, a visual description for images and video), and that text is embedded alongside your text documents. Your query matches the content of the media, not just its filename.

This dual approach means a query for a song finds audio files both by matching the sound of the music (native embedding) and by matching the transcribed lyrics (text embedding). A search for the line “uptown funk you up” surfaces the right track even when the file is named track03.mp3.

Why reranking is required

Relevance scores from text search and media search are produced by different models on different scales. A text reranker score of 0.7 and a media cosine similarity of 0.7 do not mean the same thing and cannot be sorted into a single list.

Captain solves this with reranker-informed pipeline weighting: the text reranker’s scores on media descriptions determine how much weight each modality gets in the final ranking. Without the reranker there is no way to produce a meaningful cross-modal ranking, so multimodal collections always rerank. Captain applies it automatically; you do not need to set rerank=true yourself.

Text-only collections

If your collection contains only text documents, multimodal search adds zero overhead. Captain automatically detects whether a collection has media content and skips the multimodal pipeline entirely for text-only collections.

Querying Multimodal Collections

Example: Multimodal Search

1
import requests
2
import uuid
3
4
BASE_URL = "https://api.runcaptain.com"
5
API_KEY = "your_api_key"
6
headers = {
7
    "Authorization": f"Bearer {API_KEY}",
8
    "Content-Type": "application/json",
9
    "Idempotency-Key": str(uuid.uuid4())
10
}
11
12
response = requests.post(
13
    f"{BASE_URL}/v2/collections/my_media_library/query",
14
    headers=headers,
15
    json={
16
        "query": "product demo with live dashboard",
17
        "rerank": True,
18
        "top_k": 10
19
    },
20
    timeout=120.0
21
)
22
23
data = response.json()
24
for result in data["search_results"]:
25
    modality = result["metadata"].get("modality", "text")
26
    print(f"[{modality}] {result['filename']} | score: {result['score']:.3f}")
27
    # For video/audio results, the segment time range is in metadata.startSec /
28
    # metadata.endSec (seconds).
29
    meta = result["metadata"]
30
    if meta.get("startSec") is not None:
31
        print(f"  segment {meta['startSec']:.0f}s-{meta['endSec']:.0f}s")
32
    print(f"  {result['content'][:100]}")

Response:

1
{
2
  "success": true,
3
  "search_results": [
4
    {
5
      "score": 0.95,
6
      "content": "[Video: product_demo.mp4]\nIn the video, a presenter walks through the analytics dashboard showing real-time revenue metrics and user engagement charts...",
7
      "document_id": "doc_video_123",
8
      "filename": "product_demo.mp4",
9
      "uri": "s3://my-media-bucket/videos/product_demo.mp4",
10
      "chunk_index": 1,
11
      "metadata": {
12
        "modality": "video",
13
        "source": "multimodal",
14
        "startSec": 120.0,
15
        "endSec": 240.0
16
      }
17
    },
18
    {
19
      "score": 0.88,
20
      "content": "[Image: dashboard_screenshot.png]\n### Visual Description\nA screenshot of a web application dashboard with a dark theme. The main area displays a line chart of monthly revenue trending upward...",
21
      "document_id": "doc_img_456",
22
      "filename": "dashboard_screenshot.png",
23
      "uri": "s3://my-media-bucket/images/dashboard_screenshot.png",
24
      "chunk_index": 0,
25
      "metadata": {
26
        "modality": "image",
27
        "source": "multimodal"
28
      }
29
    },
30
    {
31
      "score": 0.82,
32
      "content": "The dashboard provides real-time analytics including revenue metrics, user engagement, and conversion rates...",
33
      "document_id": "doc_pdf_789",
34
      "filename": "product_docs.pdf",
35
      "uri": "s3://my-company-docs/product_docs.pdf",
36
      "chunk_index": 15,
37
      "metadata": {
38
        "modality": "text",
39
        "source": "hybrid",
40
        "pageStart": 8,
41
        "pageEnd": 8
42
      }
43
    }
44
  ],
45
  "total_results": 3,
46
  "top_k": 10,
47
  "query": "product demo with live dashboard",
48
  "execution_time_ms": 1240
49
}

Request Fields

Response Fields (per search result)

Spreadsheet row/column provenance

For spreadsheet sources (XLSX, XLS, CSV, TSV), each chunk is a structure-aware block of rows rendered as a markdown table, and carries first-class row/column provenance - the tabular analogue of pageStart/pageEnd for PDFs. This lets you ground an answer in concrete coordinates, e.g. “Sheet ‘YTD Sales’, rows 2–13, columns Year–Ytd Total Sales”.

1
{
2
  "score": 0.81,
3
  "content": "# Sheet: YTD Sales  -  section 1 (2024)\n| Year | Month | Weekly Net Sales | Ytd Total Sales |\n| --- | --- | --- | --- |\n| 2024 | Jan | 8025 | 339987 |",
4
  "document_id": "doc_xyz789",
5
  "filename": "sales.xlsx",
6
  "chunk_index": 0,
7
  "metadata": {
8
    "sheetName": "YTD Sales",
9
    "section": "section 1 (2024)",
10
    "rowStart": 2,
11
    "rowEnd": 13,
12
    "colStart": 1,
13
    "colEnd": 4,
14
    "columns": ["Year", "Month", "Weekly Net Sales", "Ytd Total Sales"],
15
    "rowRole": null
16
  }
17
}

These fields are null for non-spreadsheet sources, exactly as pageStart/pageEnd are null for spreadsheets. The chunk’s content repeats the header row, so the table is self-describing even when read in isolation.

Media segment time range

For video and audio, each result is one segment of the source file. Its time range is returned as the structured fields metadata.startSec and metadata.endSec (seconds), the time analogue of pageStart/pageEnd for documents. Read these directly; do not parse the content string.

1
{
2
  "score": 0.88,
3
  "content": "[Video: product_demo.mp4]\nThe presenter demonstrates the checkout flow...",
4
  "metadata": { "modality": "video", "startSec": 120.0, "endSec": 158.4 }
5
}

The content field’s first line is a citation label ([Video: <filename>]) followed by the VLM description and (for video) the spoken transcript. The label no longer carries the time range. startSec/endSec are null for non-media (text/PDF) results, just as pageStart/pageEnd are null for media.

If you pass rerank=false on a multimodal collection, the query still succeeds. Reranking is applied anyway, and the response carries a warnings entry telling you the flag was overridden:

1
{
2
  "success": true,
3
  "warnings": [
4
    "Reranking cannot be disabled for collections that contain multimodal content (images, video, or audio); reranking was applied."
5
  ],
6
  "search_results": [ ... ]
7
}

Text-only collections honor rerank=false as before.

Indexing Media Files

Media files are indexed through the same endpoints as text documents. No special configuration is needed-Captain automatically detects the file type and routes it through the appropriate processing pipeline.

Transcription language (video and audio)

Speech in video and audio files is transcribed with automatic language detection by default — Spanish interviews come back as Spanish text, Portuguese as Portuguese, and so on, with no configuration. Detection runs on the whole file’s audio, so it is reliable even for short clips.

When you already know the spoken language (or a file mixes languages and you want to force the dominant one), pass the optional transcription_language field on any indexing request, as an AWS Transcribe language code such as es-US, es-ES, pt-BR, or en-US:

1
response = requests.post(
2
    f"{BASE_URL}/v2/collections/consumer_interviews/index/url",
3
    headers=headers,
4
    json={
5
        "url": "https://example.com/entrevista-consumidor.mp4",
6
        "processing_type": "advanced",
7
        "transcription_language": "es-US"
8
    }
9
)

• Omit the field to auto-detect (recommended for most workloads).
• Codes are case-insensitive (es-us is accepted and canonicalized to es-US); an unsupported value is rejected with a 422 listing example codes.
• The field applies to video and audio files only; other file types in the same request ignore it.
• Each result’s metadata.transcriptLanguage reports the language the transcript was actually recognized in.
• Files indexed before language support (or while a former English-only default applied) can be re-indexed with overwrite_existing: true to regenerate their transcripts.

Index from cloud storage

1
import requests
2
3
BASE_URL = "https://api.runcaptain.com"
4
API_KEY = "your_api_key"
5
headers = {
6
    "Authorization": f"Bearer {API_KEY}",
7
    "Content-Type": "application/json"
8
}
9
10
# Index an S3 bucket containing mixed content (PDFs, images, videos, audio)
11
response = requests.post(
12
    f"{BASE_URL}/v2/collections/my_media_library/index/s3",
13
    headers=headers,
14
    json={
15
        "bucket_name": "my-media-bucket",
16
        "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
17
        "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
18
        "bucket_region": "us-east-1",
19
        "processing_type": "advanced"
20
    }
21
)
22
23
print(response.json())

Index from URL

1
# Index media files from direct URLs
2
response = requests.post(
3
    f"{BASE_URL}/v2/collections/my_media_library/index/url",
4
    headers=headers,
5
    json={
6
        "urls": [
7
            "https://example.com/product-demo.mp4",
8
            "https://example.com/screenshot.png",
9
            "https://example.com/podcast-episode.mp3"
10
        ],
11
        "processing_type": "basic"
12
    }
13
)

Upload files directly

1
# Upload files via multipart form
2
with open("meeting_recording.mp4", "rb") as video, open("notes.pdf", "rb") as pdf:
3
    response = requests.post(
4
        f"{BASE_URL}/v2/collections/my_media_library/index/file",
5
        headers={
6
            "Authorization": f"Bearer {API_KEY}",
7
        },
8
        files=[
9
            ("files", ("meeting_recording.mp4", video, "video/mp4")),
10
            ("files", ("notes.pdf", pdf, "application/pdf")),
11
        ],
12
        data={"processing_type": "basic"}
13
    )

Using Retrieved Media Chunks

The query endpoint returns retrieved chunks across all content types. Your application can pass those chunks to an LLM, cite them directly, or store chunk IDs for later metadata and relation work.

1
response = requests.post(
2
    f"{BASE_URL}/v2/collections/my_media_library/query",
3
    headers=headers,
4
    json={
5
        "query": "What was discussed about the product roadmap?",
6
        "rerank": True,
7
        "top_k": 5
8
    },
9
    timeout=120.0
10
)
11
12
data = response.json()
13
for result in data["search_results"]:
14
    print(result["chunk_id"], result["content"][:160])

Media results use labels such as [Video: meeting_recording.mp4] followed by the VLM-generated description, with the segment time range in metadata.startSec/metadata.endSec.

Limitations

• Video segments: Maximum 120 seconds per segment. Longer videos are automatically split.
• Mixed-language media: Language detection picks one dominant language per file. For video or audio that switches languages mid-file, force the dominant one with transcription_language.
• Audio segments: Maximum 80 seconds per segment. Longer audio files are automatically split.
• Format conversion: Some formats are converted at ingestion (FLAC→MP3, WebP→PNG, AVI→MP4). The original file is preserved.
• Reranking: Multimodal collections always rerank; the rerank flag is ignored for them, and a warnings entry is returned if you pass rerank=false. Text-only collections are unaffected.
• Latency: Multimodal queries take ~1-2 seconds (text embedding + media search + reranking). Text-only queries are unchanged. We are continuously optimizing query latency to make search as fast as possible without compromising accuracy.