A rank-1 LoRA adapter is only a few MB, so async GRPO can span separate HF Jobs: trainer and vLLM replicas swap adapters via a mounted Storage Bucket instead of NCCL, with a proxy adding auth, broadcasting adapter loads, and routing rollouts by KV prefix. Five instrumented runs chase the bottleneck between training and generation; packing, dropping gradient checkpointing, and raising in-flight caps cut 500 steps from 3h27m to 53min at equal reward. Tradeoffs: FUSE sync lag, retry logic, concurrency caps.
Uber replaced mobile app polling—which reached 80% of gateway traffic—with RAMEN, a server-push platform built on Server-Sent Events plus a custom sequence-number/acknowledgment protocol for at-least-once delivery. Scaling forced a rewrite from Node.js with Ringpop gossip sharding to Netty with ZooKeeper/Helix, reaching 1.5M concurrent connections and 250K messages/sec. A newer gRPC bidirectional streaming version addresses delayed acks, unidirectional transport, and text-only payload limits.
Uber recounts migrating from a monolithic codebase to a microservices SOA (500+ services) to handle rapid growth, which solved coupling and deployment problems but introduced issues of obviousness, safety, and resilience. They adopted Apache Thrift for strict cross-language service contracts and built client-publishing tooling, drawing on Netflix Hystrix and Twitter Finagle for fault tolerance. Tradeoffs include immature Thrift tooling for Python and Node, and no header support complicating authentication and tracing.
Cloudflare cut ~100TB of RAM from its Pingora Backend Router by attacking pingora-ketama's consistent-hashing rings on two fronts: packing the per-point struct from 8 bytes to 6 (25% savings, done via a raw byte array because Rust's alignment rules negate simply shrinking the index field), and reducing hashes per server by 90%, justified by a derived formula (CV_k = sqrt((N-1)/(Nk+1))) showing the last 90,000 of ~100,000 hashes bought only ~0.7% error reduction while 32-bit collisions actually made error worse at high hash counts. The tradeoffs are that fewer hashes raises the theoretical load-imbalance error margin, and changing the ring re-routes cacheable requests and would invalidate cached content, so both rings ran side by side per request and the rollout proceeded data-center by data-center to limit cache churn and blast radius.