File Size Limits

Limits by ingestion path

The size limit depends on how the file gets to Captain, not on what kind of file it is.

Cloud storage connectors cover S3, GCS, Azure Blob, Cloudflare R2, Supabase Storage, Backblaze B2, Dropbox, Google Drive, SharePoint, and OneDrive. Captain reads those files from the source, so nothing passes through an upload endpoint and no upload cap applies. The per-file-type limits below still do.

Files above 4 MB uploaded from the dashboard switch to a multipart upload automatically. You do not need to do anything differently.

Limits by file type

JSON

.json files have a hard 500 MB ceiling, checked before the file is downloaded. This applies to every JSON file Captain indexes, including ones pulled in through a cloud storage connector.

If you pass a parsing_script, the file also has to fit in the sandbox that runs your script:

A JSON file between 50 MB and 500 MB indexes fine without a parsing script. With one, it fails. See Parsers for the full parsing script contract.

Text, Markdown, CSV, YAML

.txt, .md, .csv, .tsv, .yaml, and .yml have no size limit of their own. The only ceiling is the ingestion path limit above, so 500 MB through the dashboard or a cloud storage connector, 100 MB through the direct upload endpoint.

Everything else

PDF, Office documents, images, audio, and video have no separate byte cap. Very large media files take longer to process because they are transcribed or analyzed frame by frame, but they are not rejected on size.

What happens when you go over

Upload endpoints reject the request with a 400 and name the limit:

1
{
2
  "detail": "File 'catalog.json' exceeds maximum size of 100MB"
3
}

Connector-based jobs accept the request and fail during processing. The error shows up on the job record, which you can read with GET /v2/jobs/{job_id}.

Working with larger files

1. Upload through the dashboard, not the API. The dashboard path allows 500 MB per file against 100 MB for POST /v2/collections/{name}/index/file.
2. Put the file in cloud storage and index from there. Connector-based indexing has no upload cap. Drop the file in S3, GCS, or Google Drive and point Captain at it.
3. Split very large JSON. A 600 MB JSON file will not index. Split it into per-record or per-section files, which also produces better search results than one enormous document.
4. Drop the parsing script for large JSON. If a JSON file is over 50 MB and the parser is the only thing blocking it, index it without parsing_script and Captain will index the raw JSON as text.