Remote Caching

RSConstruct supports sharing build artifacts across machines via remote caching. When enabled, build outputs are pushed to a remote store and can be pulled by other machines, avoiding redundant rebuilds.

Configuration

Add a remote URL to your [cache] section in rsconstruct.toml:

[cache]
remote = "s3://my-bucket/rsconstruct-cache"

Supported Backends

Amazon S3

[cache]
remote = "s3://bucket-name/optional/prefix"

Requires:

• AWS CLI installed (aws command)
• AWS credentials configured (~/.aws/credentials or environment variables)

The S3 backend uses aws s3 cp and aws s3 ls commands.

HTTP/HTTPS

[cache]
remote = "http://cache-server.example.com:8080/rsconstruct"
# or
remote = "https://cache-server.example.com/rsconstruct"

Requires:

• curl command
• Server that supports GET and PUT requests

The HTTP backend expects:

• GET /path to return the object or 404
• PUT /path to store the object
• HEAD /path to check existence (returns 200 or 404)

Local Filesystem

[cache]
remote = "file:///shared/cache/rsconstruct"

Useful for:

• Network-mounted filesystems (NFS, CIFS)
• Testing remote cache behavior locally

Control Options

You can control push and pull separately:

[cache]
remote = "s3://my-bucket/rsconstruct-cache"
remote_push = true   # Push local builds to remote (default: true)
remote_pull = true   # Pull from remote on cache miss (default: true)

Pull-only mode

To share a read-only cache (e.g., from CI):

[cache]
remote = "s3://ci-cache/rsconstruct"
remote_push = false
remote_pull = true

Push-only mode

To populate a cache without using it (e.g., in CI):

[cache]
remote = "s3://ci-cache/rsconstruct"
remote_push = true
remote_pull = false

How It Works

Cache Structure

Remote cache stores two types of objects:

1. Index entries at index/{cache_key}

   • JSON mapping input checksums to output checksums
   • One entry per product (source file + processor + config)

2. Objects at objects/{xx}/{rest_of_checksum}

   • Content-addressed storage (like git)
   • Actual file contents identified by SHA-256

On Build

1. RSConstruct computes the cache key and input checksum
2. Checks local cache first
3. If local miss and remote_pull = true:
   • Fetches index entry from remote
   • Fetches required objects from remote
   • Restores outputs locally
4. If rebuild required:
   • Executes the processor
   • Stores outputs in local cache
   • If remote_push = true, pushes to remote

Cache Hit Flow

Local cache hit → Restore from local → Done
       ↓ miss
Remote cache hit → Download index + objects → Restore → Done
       ↓ miss
Execute processor → Cache locally → Push to remote → Done

Best Practices

CI/CD Integration

In your CI pipeline:

# .github/workflows/build.yml
env:
  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

steps:
  - run: rsconstruct build

Separate CI and Developer Caches

Use different prefixes to avoid conflicts:

# CI: rsconstruct.toml.ci
[cache]
remote = "s3://cache/rsconstruct/ci"
remote_push = true
remote_pull = true

# Developers: rsconstruct.toml
[cache]
remote = "s3://cache/rsconstruct/ci"
remote_push = false  # Read from CI cache only
remote_pull = true

Cache Invalidation

Cache entries are keyed by:

• Processor name
• Source file path
• Processor configuration hash

To force a full rebuild ignoring caches:

rsconstruct build --force

To clear only the local cache:

rsconstruct cache clear

Troubleshooting

S3 Access Denied

Check your AWS credentials:

aws s3 ls s3://your-bucket/

HTTP Upload Failures

Ensure your server accepts PUT requests. Many static file servers are read-only.

Slow Remote Cache

Consider:

• Using a closer region for S3
• Enabling S3 Transfer Acceleration
• Using a caching proxy

Debug Mode

Use verbose output to see cache operations:

rsconstruct build -v

This shows which products are restored from local cache, remote cache, or rebuilt.