The core premise of HireOS is that users supply their own API keys rather than paying for AI access through the platform. This unlocks a different class of product: no subscription margin stacked on top of model costs, full transparency about which model actually runs, and the user's own rate limits instead of a shared pool. But it also means the routing layer is more complex than a single fetch() to one provider.
This post walks through how I designed that layer — how keys are handled safely, how the router decides which provider to call, and how fallback works when a key is invalid or a provider is down.
Why BYOK changes the architecture
In a platform-pays model, you hold one set of organisation keys, meter usage, and bill a margin on top. The architecture is simple and the keys live in one trusted place. BYOK inverts every part of that.
Now each user has their own keys, for potentially several providers, and those keys are secrets you're holding on their behalf. That forces three design decisions up front: where per-user keys are stored, which part of your system is allowed to decrypt and use them, and how the request path keeps those keys away from anything that doesn't strictly need them. Running the actual inference call from the edge — close to the user and isolated from your main application origin — made the most sense, because it keeps the secret-handling surface small and the latency low.
Key handling: storage, encryption, and scope
Keys are stored encrypted and scoped to the individual user — not in plaintext, and never shared across accounts. The principle I held to: the fewer components that can see a raw key, the better.
- Keys are persisted in an encrypted column, retrievable only for the owning user.
- The Next.js origin orchestrates the request but is kept out of the raw-key path where possible; the inference call itself runs in a Cloudflare Worker that decrypts and uses the key only for the duration of the request.
- Keys are never logged, never returned to the client after entry, and never embedded in URLs or query strings.
The mental model is "a key is borrowed for one call, used, and forgotten" — not "a key sits warm in application memory."
The provider abstraction layer
Five providers means five different request shapes, auth schemes, model identifiers, streaming formats, and error conventions. The fix is a single interface that every provider adapter implements, so the rest of the system never special-cases a vendor.
Each adapter exposes the same small surface — something like complete(), streamComplete(), countTokens(), and a normalised modelId mapping. A factory picks the right adapter from the user's selected provider. Adding a sixth provider then means writing one adapter that satisfies the interface, with zero changes to the routing or UI code. Normalising model names across providers is the unglamorous but essential part: the app speaks one vocabulary, and each adapter translates.
Routing logic: how HireOS picks a provider
Routing starts with explicit user choice — the user picks the provider and model they want. From there the router layers in practical signals: whether the selected key is present and valid, and a fallback order for when it isn't. Because the call runs at the edge, latency awareness is cheap to factor in, and cost signals can inform a default when the user hasn't expressed a preference.
The important discipline is keeping routing predictable. Users supplying their own keys want to know exactly which model ran and what it cost them — so silent provider substitution is only ever a transparent fallback, never a cost optimisation done behind their back.
Fallback and error handling
The failure modes in BYOK are different from a platform-keys setup, because the most common error is one you don't control: the user's key.
- Invalid or expired key — detect it early, return a clear, specific message ("your OpenAI key was rejected"), and don't silently fall through to a provider the user didn't choose.
- Provider 429 / 503 — back off, and only then offer a fallback provider if the user has a valid key for one.
- Error surfacing — give the user enough to act ("rate limited by Anthropic, retry shortly") without leaking internal routing details or partial secrets.
The goal is errors that the user can fix, because in BYOK the fix is usually in their hands.
Lessons from running BYOK in production
A few things I'd flag to anyone building this:
- Key validation UX is the product. Most support friction is "why isn't it working," and it's almost always a key issue. Validate on entry, not on first use.
- Transparency builds trust. Showing the exact model and a cost estimate per call turned a support burden into a selling point.
- BYOK isn't always right. It suits technical, cost-sensitive users who already have keys. For a non-technical audience, the friction of "go get five API keys" can kill activation — and a platform-pays model is the better call.