Anthropic uses Pub/Sub as the backbone of its ingestion pipeline. Because large machine learning payloads—such as high-dimensional embeddings and activations—frequently exceed multiple megabytes and can scale up to 100 MB, Anthropic designed a dual-path pipeline to optimize network throughput and cost-efficiency.

Producers serialize events into Protocol Buffers and apply zstd compression. If the compressed payload is under 8 MB, the client publishes the message directly to a Pub/Sub topic. If the payload is 8 MB or larger, the client writes the compressed bytes directly to a Cloud Storage bucket and publishes a lightweight JSON pointer notification containing the object URI through Pub/Sub. The downstream consumer uses this notification to fetch and decompress the payload from Cloud Storage. By reusing the pre-compressed bytes for the Cloud Storage upload, the pipeline avoids redundant compression cycles, keeping the messaging broker entirely unblocked.

One of Anthropic’s top priorities was reducing latency. When they profiled one of their consumer pipelines, they found that 88% of their latency was caused by Pub/Sub’s synchronous pull processing mode. Anthropic’s consumer fleet comprised 60 pods running ten parallel coroutines each—creating 600 concurrent connections on one subscription. When queues were empty, consumer threads were blocked for the duration of the long-poll timeout (ranging from 8 to 18 seconds).

To resolve this bottleneck, Anthropic migrated to StreamingPull. By establishing persistent, bidirectional gRPC connections, Pub/Sub pushed messages directly to workers as soon as they were published. This change eliminated synchronous polling overhead, dropping consumer latency to nearly zero.

Additionally, Anthropic’s Python services use the Trio concurrency library. Pub/Sub’s Python client library uses asyncio. Anthropic engineers built custom adapters to bridge integrate their Trio runtime using bounded queues and proper backpressure to prevent slow publishers from blowing out memory.