THE COLLECTION

Library

Everything you’ve found. Nothing you need to save twice.

28 articles
Browse at your own pace.

Laya — 33ms Multilingual System 1 Decision Engine

The author introduces Laya, an open-source family of bidirectional encoder models that answer typed questions (choice, ordinal score, boolean) with calibrated probabilities in roughly 33 ms, arguing generative LLMs are overkill for high-volume triage and routing. Self-reported benchmarks claim speed, calibration, and cost advantages over the proprietary Jev API. Acknowledged tradeoffs: choice questions degrade beyond ~20 options, base checkpoints are near-random zero-shot and require fine-tuning, and temperature calibration is needed.

How we used DSPy to turn AI evaluations into better responses in Dash chat

Dropbox built trajectory-based LLM-as-judge evaluations for its Dash chat agent, then used DSPy's GEPA and MIPROv2 optimizers in two stages: calibrating judges against a small human-labeled set, then optimizing the agent's system prompt via offline replay of historical chats. Reported gains include 26% fewer incomplete answers, 13% fewer missed key aspects, and 5.4% lower token usage. The tradeoffs: automation requires strict guardrails, and weak evaluation signals risk brittle improvements.

How well do agents use test/verification techniques?

An empirical eval of 26 prompt conditions (TDD, fuzzing, property-based testing, formal methods, and testing skills) given to coding agents implementing Zstd in Rust, ~80 runs each. Nothing beat the default no-instructions baseline; agents apply techniques superficially—vacuous proofs, trivial random tests—while TDD and popular testing skills underperformed. A brief hand-written skill nudging risky-area checks and structured randomization scored best, suggesting expert guidance matters more than naming techniques.

Saving another 100TB of RAM with math (and Rust)

Cloudflare cut ~100TB of RAM from its Pingora Backend Router by attacking pingora-ketama's consistent-hashing rings on two fronts: packing the per-point struct from 8 bytes to 6 (25% savings, done via a raw byte array because Rust's alignment rules negate simply shrinking the index field), and reducing hashes per server by 90%, justified by a derived formula (CV_k = sqrt((N-1)/(Nk+1))) showing the last 90,000 of ~100,000 hashes bought only ~0.7% error reduction while 32-bit collisions actually made error worse at high hash counts. The tradeoffs are that fewer hashes raises the theoretical load-imbalance error margin, and changing the ring re-routes cacheable requests and would invalidate cached content, so both rings ran side by side per request and the rollout proceeded data-center by data-center to limit cache churn and blast radius.