Put a gateway in front of a federation and someone will eventually ask for the same four things at once: team A sees these tools, gets this allowance, cannot flood team B, and shows up separately in the audit trail. Every one of those was already possible in fold. Assembling them was the problem: four unrelated mechanisms, and the word tenant appearing nowhere in the config, the audit stream, or the metrics. This post is about what it took to make that one object, and the two decisions inside it that were not obvious.
What a tenant is not#
One line shapes everything else: a tenant groups principals, it does not authenticate them. It is derived from the verified principal, after authentication has already happened. It is never presented alongside a token, and it is never a trust anchor. Policy remains the sole authority on what may be invoked.
That sounds like a restriction and it is really a safety property. The moment a tenant id can be asserted by a caller, it becomes a credential, and a weak one: a string that grants an allowance and a view of the federation. Deriving it instead means the worst a confused tenant mapping can do is give someone the wrong budget. It can never give them authority they did not already authenticate into.
The feature is additive by construction. Declare no tenants and a deployment behaves exactly as it did before tenancy existed, which is the only honest way to ship a cross-cutting dimension into something already in production.
One bucket, not ten#
This is the distinction that motivates the whole object. fold already had server.rateLimit.perPrincipalPerMinute, which gives each person a bucket. Ten agents on one team therefore hold ten allowances between them, and "team A cannot flood team B" is simply not what that setting says. A tenant's rateLimit is one bucket shared by every principal in it. That is the sentence people thought they were already buying.
"tenants": [
{
"id": "acme",
"subjects": { "claims": { "org_id": "acme-prod" } }, // the shape policy rules already use
"budget": { "period": "month", "upstreamCalls": 500000 },
"rateLimit": { "requestsPerMinute": 2000 }, // one bucket for the whole tenant
"upstreams": ["billing", "crm"] // optional; all upstreams if omitted
}
]subjects reuses the selector shape policy rules already use (groups, subs, issuers, claims), because an operator who has written one policy rule should not have to learn a second grammar to write a tenant. A tenant with no selector would capture every caller, so it is rejected at load rather than quietly swallowing the federation.
Two orderings, deliberately opposite#
Once an allowance can sit at three scopes (upstream, tenant, server) you have to decide what order to consult them in. fold uses opposite orders for the two mechanisms, and the asymmetry is the interesting part.
- Budgets charge narrowest-first: upstream, then tenant, then server. A request that is going to be refused must not spend a wider allowance on its way to the refusal. Charge the server budget first and a tenant that is over its own limit still burns the organisation's monthly allowance every time it is turned away.
- Rate limits check widest-first: global, then tenant, then per-principal. A flood should be refused before it costs any routing work. The cheapest possible answer to a stampede is the one you can give without looking anything up.
Stated together they look inconsistent. They are not: both orderings minimise what a refusal costs, and the two mechanisms simply have different costs to minimise. A budget's cost is allowance, so consult the allowance you are least willing to waste last. A rate limit's cost is work, so consult the check that is cheapest to fail first.
The two also refuse different things, which is why both exist. A rate limit answers "how fast" and then forgets; a sliding window smooths a burst and lets it go. A budget answers "how much this month" and remembers, accumulating until the period rolls over. Smoothing a burst is the correct behaviour for protecting a fragile upstream and the wrong one for bounding a month's spend.
Visibility happens at the fan-out, not the result#
upstreams bounds which upstreams a tenant's callers reach, and it is evaluated before policy. Filtering at the fan-out rather than the result has consequences that are easy to miss and worth stating:
- An upstream outside the subset is never asked. It costs no request, no budget, and no partial-failure entry when it happens to be down. A tenant is not exposed to the health of infrastructure it cannot reach.
- A named invocation against one is refused ahead of the policy engine, with
-32042. tasks/*is the deliberate exception. Those answer "no upstream owns that id" rather than issuing a denial, matching the posture that path already takes for another principal's task. Where a refusal would confirm that something exists, it does not refuse.- A viewer's console shows their tenant's federation, not the operator's. That closes the one place a dashboard could otherwise show a customer the shape of a topology its own traffic is refused.
Ambiguity is refused, not resolved#
A principal belongs to at most one tenant. Where two selectors overlap, fold refuses rather than picking a winner. The alternative is precedence, and precedence in this position is a trap: assigning a caller by declaration order hands them another tenant's allowance and another tenant's view of the federation on the day somebody reorders a list. That is a config edit with no visible blast radius and a real one.
Some overlap is catchable at validation, and is caught there. Some is only decidable against a real principal, because two selectors can collide for some callers and not others; that is caught at request time and refused there instead. A principal matching no tenant simply has none, and is governed exactly as it was before tenancy existed.
Ten thousand tenants, 97 nanoseconds#
Tenant resolution runs on every request, so it cannot be a scan. Two selector shapes cover what a per-customer document actually repeats: one claim equalling one value, or one group. Both are indexed at snapshot time, and the two indexes are keyed from opposite sides: the claim index by what the tenant requires, the group index by what the principal holds.
Ten thousand tenants resolve in 97 ns with zero allocations, which is the same cost as ten. A linear scan was 450 µs at that size. Compound selectors still scan, so keep those in the tens rather than the thousands. Methodology and the full table are on the benchmarks page, including the hardware and the caveats.
New names, not new labels#
With tenants configured, the dimension shows up in both observability surfaces: tenant on every audit event its principals produce, denials included, and two new metrics, fold_tenant_requests_total{tenant,outcome} and fold_tenant_upstream_calls_total{tenant}. The second counts the unit a tenant budget is charged in, so an allowance can be watched being spent rather than discovered exhausted.
Those are new metric names rather than a tenant label added to the existing ones, and that was not a stylistic choice. Label sets are frozen by fold's compatibility contract. Adding a label to an existing series changes its identity in Prometheus and breaks every dashboard, alert, and recording rule built on it. A feature that silently rewrites an operator's alerting is not additive, whatever the config file says.
The transferable lesson#
Multi-tenancy in a proxy is usually described as an isolation feature. Most of the work here was not isolation, it was deciding what a tenant is allowed to be: not a credential, not a precedence rule, not a new label on an existing series. Each of those three would have been the shorter implementation, and each would have handed an operator a sharp edge pointing at something they already depended on.
The general shape: when you add a dimension to a system that is already running, the interesting constraints are almost never in the new code path. They are in what the new dimension is forbidden from touching.
Try it#
Tenants are reloadable, unlike server.budget, because customers sign up on their own schedule and a reload has to be able to add one. The full field reference is at docs.fold.run/tenancy; budgets and metering are at /consumption; the source is on GitHub under Apache-2.0.