Llm

Google’s TurboQuant compresses LLM KV caches and embedding vectors to 3-4 bits with no training and no codebook, and still beats methods that need both. The clever part is not the quantizer, it is what it refuses to store. Here is the intuition, with runnable NumPy you can paste and check.

I just shipped llm-relay, a small OpenAI-compatible LLM gateway in pure Go (zero third-party dependencies). The interesting part is the failover: when a provider is rate-limited or down, it switches to the next one BEFORE any bytes reach the client, so the caller never sees half a stream and then a different model. Here is how it works, with the real code.