Exio

What it does

Exio aims to work with unmodified S3 clients: the AWS CLI, the main AWS SDKs, rclone, mc, s3cmd and Elixir's ex_aws_s3. Where S3's behaviour can be observed (status codes, error codes, headers, XML shape, listing order, ETag values), Exio reproduces it. Operations it does not support fail with an explicit 501 NotImplemented rather than doing something approximate.

The supported surface includes buckets, objects with ranges and conditional requests, multipart upload, copy, bulk delete, presigned URLs, tagging, CORS and bucket versioning. It also has full SigV4 signing, including the streamed chunked uploads and trailing checksums that current AWS SDKs send by default. Without those, mainstream clients cannot upload at all.

Around the S3 API sit the parts needed to run it: users with roles and per-bucket grants, access keys owned by users, TOTP second factors, a JSON admin API so every administrative task can be scripted, an audit log, and backups that stream while the server keeps running.

How it is built

A few design choices carry most of the weight:

• Each bucket is a self-contained directory with its own SQLite index. There is no global bucket table. A bucket can be backed up or moved by copying its directory, and Exio notices buckets appearing or leaving on disk without a restart.
• Object data files are immutable. A write creates a new file and then atomically repoints the key in the index. A partly written object is never visible, readers of the old version are not disturbed, and a crash leaves at worst an orphan file to sweep up.
• Multipart completion copies nothing. A finished upload is stored as its parts, linked into place, so completing a very large upload costs one link per part rather than a full rewrite.
• Every request and response body is streamed, so memory per request stays bounded whatever the object size.
• Small objects live inside the index, so many tiny objects do not exhaust the filesystem's inodes.

The S3 endpoint and the console listen on separate ports. S3 routes on method and query string in ways a web framework's router cannot express, and the S3 endpoint must never parse bodies or set sessions.

How it is checked

Compatibility is the product, so it is tested from several directions. The tests include the AWS signature test vectors, property tests of the chunked-upload decoder, real client programs run against the server, and the Ceph s3-tests conformance suite against the built container image. The concurrency protocols that matter most (bucket lifecycle, the instance lock, multipart upload and the versioned writer) are model-checked with ExTLA. The specification, tests and code are kept aligned with Surfex, and CI builds use Depdep.

The specification is the source of truth. Behaviours that were written from knowledge of S3 but not yet confirmed were tracked and settled in a dedicated validation pass against the AWS documentation and the conformance suite.

Status

Exio is in active development and ships as a container image with a host-side control script. Versions are still 0.x. Some deliberate gaps remain: there is no clustering or replication, no IAM-style policy engine, and encryption at rest is designed for but not yet built.