CER-0334
Amazon S3 bills for every API request — LIST, HEAD, GET, PUT, COPY, POST, and DELETE — independently of storage charges. Workloads originally designed for locally-provisioned storage, where listing a directory, checking a file's existence, or writing a single record is effectively free, carry those assumptions into S3 and convert each step into a billed HTTP request. At scale, request costs can rival or even exceed storage costs, yet they are routinely overlooked by cost-optimization efforts that focus on storage class selection and data transfer.
The waste manifests on both sides of the I/O path. On the read and coordination side, applications generate LIST and HEAD storms: legacy commit protocols recursively list output directories to discover what tasks wrote, query engines re-enumerate partitions on every execution, and consumers poll a prefix on a timer to detect new data. On the write side, metrics, event, and log pipelines issue one small PUT per record instead of buffering and flushing in batches, so PUT volume scales linearly with input rate. Rename-based commit protocols compound the problem because S3 has no native rename — each rename is implemented as a COPY followed by a DELETE, doubling the request count per output file.
The root cause is an architectural mismatch: the application treats S3 as a filesystem it can list, stat, and rename cheaply, when S3 is an object store that charges per HTTP call. Fixing the problem requires shifting coordination, state tracking, and batching into the application layer so that S3 serves bytes rather than acting as a coordination mechanism.
S3 request costs are billed per 1,000 operations, with rates that vary by operation type:
Several characteristics of this billing model amplify the cost of filesystem-oriented workloads: