Cloud Storage Credentials Guide

This guide will help you obtain the necessary credentials to use Captain with your cloud storage provider.

AWS S3 Credentials

To index files from Amazon S3 buckets, you’ll need an AWS Access Key ID and Secret Access Key.

Step 1: Navigate to IAM Security Credentials

1. Log in to the AWS Console
2. Click on your account name in the top-right corner
3. Select Security credentials from the dropdown menu

Step 2: Create Access Key

1. Scroll down to the Access keys section
2. Click the Create access key button

Step 3: Retrieve Your Credentials

1. Your Access Key ID and Secret Access Key will be displayed
2. Important: This is the only time you can view the Secret Access Key
3. Click Show to reveal the Secret Access Key
4. Copy both the Access Key ID and Secret Access Key to a secure location
5. Optionally, download the .csv file for safekeeping

Best Practices for AWS Access Keys

• Never store your access key in plain text, in a code repository, or in code
• Disable or delete access keys when no longer needed
• Enable least-privilege permissions
• Rotate access keys regularly

Required IAM Permissions

Your AWS access key needs the following permissions to work with Captain:

For read-only access to S3 buckets:

1
{
2
  "Version": "2012-10-17",
3
  "Statement": [
4
    {
5
      "Effect": "Allow",
6
      "Action": [
7
        "s3:GetObject",
8
        "s3:ListBucket"
9
      ],
10
      "Resource": [
11
        "arn:aws:s3:::your-bucket-name",
12
        "arn:aws:s3:::your-bucket-name/*"
13
      ]
14
    }
15
  ]
16
}

Replace your-bucket-name with your actual S3 bucket name.

Google Cloud Storage Credentials

To index files from Google Cloud Storage buckets, you’ll need a Service Account JSON key.

Step 1: Navigate to Service Accounts

1. Go to Google Cloud Console
2. Navigate to IAM & Admin → Service Accounts
3. Click Create Service Account

Step 2: Create Service Account

1. Enter a Service account name (e.g., captain-storage-access)
2. Add a Service account description (optional but recommended)
3. The Service account ID will be auto-generated
4. Click Create and continue

Step 3: Grant Permissions

Under Grant this service account access to project, choose the appropriate role based on your needs:

For read-only access to buckets/objects:

• Role: Storage Object Viewer (roles/storage.objectViewer)

For read/write access:

• Role: Storage Object Admin (roles/storage.objectAdmin)

For full bucket management:

• Role: Storage Admin (roles/storage.admin)

Click Continue → Done

Step 4: Create and Download JSON Key

1. You’ll now see your new service account in the list
2. Click on the service account name
3. Navigate to the Keys tab
4. Click Add Key → Create New Key
5. Select JSON as the key type
6. Click Create

The JSON key file will automatically download to your computer. This file contains your service account credentials.

JSON Key File Format

Your downloaded JSON key will look like this:

1
{
2
  "type": "service_account",
3
  "project_id": "your-project-id",
4
  "private_key_id": "abc123...",
5
  "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
6
  "client_email": "captain-storage-access@your-project.iam.gserviceaccount.com",
7
  "client_id": "123456789",
8
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
9
  "token_uri": "https://oauth2.googleapis.com/token",
10
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
11
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/..."
12
}

Using the Service Account with Captain

When using Captain with Google Cloud Storage, you’ll typically need to:

1. Store the JSON key file securely on your server
2. Set the GOOGLE_APPLICATION_CREDENTIALS environment variable to point to the JSON file path:

$
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/your-service-account-key.json"

Or load it programmatically in your application.

Best Practices for Service Account Keys

• Service account keys could pose a security risk if compromised
• Never commit service account keys to version control
• Store keys securely using secret management services
• Rotate keys regularly
• Delete unused service accounts and keys
• Consider using Workload Identity Federation for more secure authentication (advanced)
• Google automatically disables service account keys detected in public repositories

Using Credentials with Captain

Once you have your credentials, you can use them with Captain’s indexing endpoints:

AWS S3 Example

1
import requests
2
from urllib.parse import quote
3
4
aws_secret = "your_aws_secret_key"
5
aws_secret_encoded = quote(aws_secret, safe='')
6
7
response = requests.post(
8
    "https://api.runcaptain.com/v1/index-s3",
9
    data={
10
        'database_name': 'my_database',
11
        'bucket_name': 'my-s3-bucket',
12
        'aws_access_key_id': 'AKIAIOSFODNN7EXAMPLE',
13
        'aws_secret_access_key': aws_secret_encoded,
14
        'bucket_region': 'us-east-1',
15
        'api_key': 'cap_dev_...',
16
        'organization_id': '01999eb7-...'
17
    }
18
)

Google Cloud Storage Example

1
import requests
2
3
# Load service account JSON
4
with open('path/to/service-account-key.json', 'r') as f:
5
    service_account_json = f.read()
6
7
response = requests.post(
8
    "https://api.runcaptain.com/v1/index-gcs",
9
    data={
10
        'database_name': 'my_database',
11
        'bucket_name': 'my-gcs-bucket',
12
        'service_account_json': service_account_json,
13
        'api_key': 'cap_dev_...',
14
        'organization_id': '01999eb7-...'
15
    }
16
)

Security Reminders

• Never share your credentials in public forums, chat messages, or email
• Never commit credentials to version control systems (Git, SVN, etc.)
• Use environment variables or secure secret management services in production
• Rotate credentials regularly to minimize security risks
• Apply least-privilege permissions - only grant the minimum access needed
• Monitor credential usage through AWS CloudTrail or Google Cloud Audit Logs

Troubleshooting

AWS Issues

Error: “Invalid AWS credentials”

• Verify your Access Key ID and Secret Access Key are correct
• Check that the access key is active in the IAM console
• Ensure your IAM user/role has the necessary S3 permissions

Error: “Access Denied”

• Verify your IAM permissions include s3:GetObject and s3:ListBucket
• Check bucket policies and ensure they allow your IAM user/role
• Verify the bucket region matches the bucket_region parameter

Google Cloud Issues

Error: “Invalid service account credentials”

• Verify the JSON key file is valid and not corrupted
• Check that the service account is enabled
• Ensure the service account has the necessary Storage permissions

Error: “Permission denied”

• Verify the service account has the appropriate Storage role
• Check that the bucket exists and the service account has access
• Review IAM permissions in the Google Cloud Console

Need Help?

If you encounter issues obtaining or using your cloud storage credentials, contact Captain support:

• Email: support@runcaptain.com
• Phone: +1 (260) CAP-TAIN