SaaS Architecture for AI Products: Multi-Tenancy, Usage Metering, and Model Cost Controls

Key Takeaways

• AI breaks the oldest assumption in SaaS economics, that one more request costs almost nothing. Every model call has a variable cost, so tenancy, metering, and cost control are one system.
• For most B2B AI products, a shared schema with a tenant_id column and Postgres row-level security is the right default. Schema and database per tenant solve specific enterprise or compliance problems.
• Row-level security only protects you if the application role does not own the tables and does not have BYPASSRLS, the policy fails closed when no tenant is set, and tenant context is set per transaction with set_config(..., true) so it survives connection pooling.
• Tenant isolation has to follow the request everywhere it goes: background jobs, caches, vector indexes, and the prompt itself. AI leaks usually happen in retrieval and caching.
• Treat usage as money. Record a usage event with an idempotency key in the same transaction as the work, keep an append-only ledger for credits, and push aggregates to billing (for example, Stripe Meters, which deduplicates on an identifier for at least 24 hours) from an outbox.
• Model cost is a formula you can control: uncached input, cache writes, cache reads, and output tokens, each at its own price. Prompt caching, model routing, token caps, and batch APIs (50 percent off at both Anthropic and OpenAI) move it far more than prompt tweaks.
• Ship guardrails before scale: per-tenant budgets with reserve-then-settle accounting, token-bucket rate limits, a kill switch per feature and per tenant, and traces that tag every model call with tenant, feature, model, prompt version, and cost.

Introduction: Why AI Changes the SaaS Cost Equation

Classic SaaS architecture grew up around a comfortable fact. Once you paid for servers, the marginal cost of one more page view, one more API call, or one more report was close to zero. Pricing could be per seat, gross margins sat comfortably high, and a heavy user was mostly a happy user.

AI features change that. Every model call is priced in tokens, and the bill scales with what users do, not how many there are. One power user can cost more than their subscription, and a runaway background loop can burn a month of margin overnight.

AI also makes multi-tenancy harder. Tenant data flows through embedding pipelines, vector indexes, caches, conversation memory, tool calls, and context windows, and each is a place where one customer's data can end up in another customer's answer.

This article is a reference architecture for multi-tenant AI SaaS in 2026, focused on the decisions that are expensive to change later: isolation, tenant context, metering, pricing, and model cost control. The code is TypeScript and SQL. Provider prices and APIs were checked against vendor documentation in October 2026; they change often, so check them again before you commit to a pricing model.

If you have a working prototype and are planning the path to production, our article on moving from AI proof of concept to a scalable product covers the process side. This article is about the architecture underneath.

Part 1: The Reference Architecture

Before going deep on any one component, it helps to see the whole system. A multi-tenant AI SaaS has the same bones as any SaaS application, plus three additions: an AI gateway that every model call passes through, a metering pipeline that turns work into billable usage, and a retrieval layer that has to respect tenant boundaries as strictly as the main database does.

Control plane and data plane

The control plane holds everything about tenants rather than tenant data: organizations, users, roles, SSO configuration, API keys, plans, budgets, and kill switches. It answers "who is this, which tenant are they acting for, and what may they do?" The data plane holds tenant data and does tenant work: the application database, object storage, vector indexes, job queues, and the AI gateway. Every data plane component receives a tenant context from the control plane and enforces it. Keeping the planes separate means an enterprise customer's dedicated database is a data plane move, and a second model provider is a gateway change.

The AI gateway is not optional

The single most important architectural decision in this article is that no application code calls a model provider directly. Every model call goes through one internal service or library, the AI gateway, which:

1. Resolves the tenant, feature, and user for the call.
2. Checks the kill switch, rate limit, and budget for that tenant and feature.
3. Chooses a model based on the task and the tenant's plan.
4. Applies token caps and prompt caching.
5. Calls the provider with timeouts, retries, and fallbacks.
6. Records exact token usage and cost, emits a usage event, and writes a trace.

Without a gateway, each concern is reimplemented, inconsistently, in every feature. With one, cost control is a configuration change instead of a code audit. It can start as a library inside your monolith.

Part 2: Choosing a Tenancy Model

There are three classic ways to isolate tenant data in a relational database, plus a hybrid that most mature products end up with.

The practical recommendation is to start with shared schema plus RLS, and build tenant-aware connection routing from day one even if every tenant routes to the same database. That routing function is the seam that lets you move one tenant to a dedicated database later without touching application code. Note that your database's tenancy model does not automatically apply to your vector store, caches, or object storage. Part 3 covers those.

Row-level security that actually isolates

Postgres row-level security (RLS) attaches policies to a table so every query, including ones a developer forgets to filter, sees only the rows the policy allows. When RLS is enabled and no policy grants access, Postgres applies default deny. Three rules make it hold up in practice:

• The application role must not own the tables. Owners normally bypass RLS. FORCE ROW LEVEL SECURITY applies policies to the owner too, but the cleaner pattern is an owner role for migrations and a separate role for the app.
• The application role must not be a superuser or have BYPASSRLS. Both skip every policy, and managed Postgres services often hand out a default admin role with one or both.
• Set the tenant per transaction, not per session. Behind a transaction-mode pooler, consecutive transactions on one connection can belong to different tenants. set_config('app.tenant_id', $1, true) is local to the transaction and disappears at commit or rollback.

Here is a complete, working setup:

-- Roles: migrations run as app_owner; the application connects as app_user.
CREATE ROLE app_owner NOLOGIN;
CREATE ROLE app_user LOGIN NOSUPERUSER NOBYPASSRLS;

CREATE SCHEMA app AUTHORIZATION app_owner;
GRANT USAGE ON SCHEMA app TO app_user;

CREATE TABLE app.tenants (
  id         uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  name       text NOT NULL,
  plan       text NOT NULL DEFAULT 'starter',
  created_at timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE app.documents (
  tenant_id  uuid NOT NULL REFERENCES app.tenants(id),
  id         uuid NOT NULL DEFAULT gen_random_uuid(),
  title      text NOT NULL,
  body       text NOT NULL,
  created_at timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (tenant_id, id)
);
ALTER TABLE app.tenants   OWNER TO app_owner;
ALTER TABLE app.documents OWNER TO app_owner;

-- Returns NULL when no tenant is set, so policies fail closed.
CREATE FUNCTION app.current_tenant() RETURNS uuid
LANGUAGE sql STABLE AS $$
  SELECT nullif(current_setting('app.tenant_id', true), '')::uuid
$$;
ALTER FUNCTION app.current_tenant() OWNER TO app_owner;

ALTER TABLE app.documents ENABLE ROW LEVEL SECURITY;
ALTER TABLE app.documents FORCE ROW LEVEL SECURITY;

-- The subselect lets the planner evaluate the tenant once per query.
CREATE POLICY tenant_isolation ON app.documents
  USING      (tenant_id = (SELECT app.current_tenant()))
  WITH CHECK (tenant_id = (SELECT app.current_tenant()));

GRANT SELECT ON app.tenants TO app_user;
GRANT SELECT, INSERT, UPDATE, DELETE ON app.documents TO app_user;

The tenant-first primary key keeps index lookups tenant-scoped. USING governs which rows are visible; WITH CHECK governs which rows can be written, so tenant A cannot insert a row labeled tenant B. And nullif(..., '') matters because a custom setting set earlier in a session reads back as an empty string, not NULL, after the transaction ends.

Prove the isolation with a test. This script runs as a superuser in a test database and switches to the application role:

INSERT INTO app.tenants (id, name) VALUES
  ('00000000-0000-0000-0000-00000000000a', 'Tenant A'),
  ('00000000-0000-0000-0000-00000000000b', 'Tenant B');

-- Seed through the application role so the policies apply to the insert too.
BEGIN;
SET LOCAL ROLE app_user;
SELECT set_config('app.tenant_id', '00000000-0000-0000-0000-00000000000a', true);
INSERT INTO app.documents (tenant_id, title, body)
VALUES (app.current_tenant(), 'A plan', 'Tenant A only');
COMMIT;

BEGIN;
SET LOCAL ROLE app_user;
SELECT set_config('app.tenant_id', '00000000-0000-0000-0000-00000000000b', true);
SELECT count(*) AS visible_to_b FROM app.documents;          -- expect 0
INSERT INTO app.documents (tenant_id, title, body)            -- expect an RLS error
VALUES ('00000000-0000-0000-0000-00000000000a', 'spoof', 'x');
ROLLBACK;

BEGIN;
SET LOCAL ROLE app_user;
SELECT count(*) AS visible_without_tenant FROM app.documents; -- expect 0
COMMIT;

In CI, this catalog query catches the most common regression, a new table nobody protected:

-- Tables with a tenant_id column but RLS not enabled or not forced
SELECT c.relnamespace::regnamespace AS schema, c.relname AS table_name,
       c.relrowsecurity AS rls_enabled, c.relforcerowsecurity AS rls_forced
FROM pg_class c
JOIN pg_attribute a ON a.attrelid = c.oid AND a.attname = 'tenant_id' AND NOT a.attisdropped
WHERE c.relkind IN ('r', 'p')
  AND c.relnamespace = 'app'::regnamespace  -- schemas that hold tenant data
  AND (NOT c.relrowsecurity OR NOT c.relforcerowsecurity);

What RLS does not do

Referential integrity checks (unique constraints, foreign keys) bypass row security, so a constraint error can reveal that a value exists in another tenant. Keep unique constraints tenant-scoped (UNIQUE (tenant_id, email)). Views run with the owner's privileges unless created with security_invoker = true (Postgres 15 and later), and SECURITY DEFINER functions run as their owner. And RLS does nothing for data you copy into a search index, a cache, or a prompt.

Tenant routing: the seam for dedicated databases

If you need stronger separation later, a routing seam makes it manageable. Every data access goes through one function that picks the database for a tenant and sets the tenant for the transaction:

// tenant-db.ts
import { Pool, PoolClient } from "pg";

const pooled = new Pool({ connectionString: process.env.POOLED_DATABASE_URL, max: 20 });
const dedicated = new Map<string, Pool>();

// Reads control.tenant_placement; returns a connection string or null. Cache it briefly.
declare function dedicatedUrlFor(tenantId: string): Promise<string | null>;

async function poolFor(tenantId: string): Promise<Pool> {
  const url = await dedicatedUrlFor(tenantId);
  if (!url) return pooled;
  if (!dedicated.has(url)) dedicated.set(url, new Pool({ connectionString: url, max: 5 }));
  return dedicated.get(url)!;
}

/** Runs fn in a transaction with the tenant set, on the right database. */
export async function withTenant<T>(tenantId: string, fn: (db: PoolClient) => Promise<T>): Promise<T> {
  const client = await (await poolFor(tenantId)).connect();
  try {
    await client.query("BEGIN");
    await client.query("SELECT set_config('app.tenant_id', $1, true)", [tenantId]);
    const result = await fn(client);
    await client.query("COMMIT");
    return result;
  } catch (err) {
    await client.query("ROLLBACK");
    throw err;
  } finally {
    client.release();
  }
}

Dedicated databases keep the same schema, policies, and app.tenant_id setting, so there is one code path, one migration set, and one test suite. A dedicated database becomes an operational choice, not a different product.

Part 3: Tenant Context Propagation

RLS protects one database. An AI product moves tenant data through many more places: HTTP handlers, queues, workers, caches, embedding pipelines, vector indexes, and the model's context window. The tenant has to be an explicit, verified value at every hop, and every store has to be scoped by it.

In the API: resolve once, carry everywhere

Resolve the tenant once per request from a verified credential (session, signed token, or API key), never from a header the client controls, and carry it in request-scoped context. In Node.js, AsyncLocalStorage does this without threading a parameter through every function:

// context.ts
import { AsyncLocalStorage } from "node:async_hooks";

export type TenantContext = {
  tenantId: string;
  actorId: string;            // user id or api key id
  actorType: "user" | "api_key" | "system";
  plan: "starter" | "pro" | "enterprise";
  requestId: string;
};

const als = new AsyncLocalStorage<TenantContext>();

export function runWithContext<T>(ctx: TenantContext, fn: () => Promise<T>): Promise<T> {
  return als.run(ctx, fn);
}

export function ctx(): TenantContext {
  const c = als.getStore();
  if (!c) throw new Error("No tenant context: refusing to continue");
  return c;
}

An authentication middleware verifies the credential, builds this context once, and calls runWithContext(context, () => next()). ctx() throws when there is no context, so code that touches tenant data outside one fails loudly instead of running as a default tenant.

In background jobs: the payload carries the tenant, the worker re-establishes it

Queues are where tenant context most often gets lost. Make tenantId a required field of every job envelope, and wrap every handler so it re-creates the context and transaction before doing work:

// jobs.ts
import { runWithContext } from "./context";
import { withTenant } from "./tenant-db";
import type { PoolClient } from "pg";

export type JobEnvelope<T> = {
  tenantId: string;
  actorId: string;
  requestId: string;       // ties the job back to the request trace
  idempotencyKey: string;  // the same job delivered twice must not double-bill
  payload: T;
};

export function tenantJob<T>(handler: (db: PoolClient, payload: T, job: JobEnvelope<T>) => Promise<void>) {
  return async (job: JobEnvelope<T>) => {
    const plan = await lookupActivePlan(job.tenantId); // throws if tenant is suspended or deleted
    await runWithContext(
      { tenantId: job.tenantId, actorId: job.actorId, actorType: "system", plan, requestId: job.requestId },
      () => withTenant(job.tenantId, (db) => handler(db, job.payload, job)),
    );
  };
}

declare function lookupActivePlan(tenantId: string): Promise<"starter" | "pro" | "enterprise">;

The lookupActivePlan check matters because jobs outlive requests: a tenant can be suspended or deleted between enqueue and execution.

In the vector store: scope retrieval by tenant, always

Retrieval is where cross-tenant leaks are most likely and hardest to detect, because leaked content is paraphrased into an answer rather than shown as a row. OWASP lists "Vector and Embedding Weaknesses" as LLM08 in its 2025 Top 10 for LLM applications. With pgvector, the simplest strong option is to keep embeddings under the same RLS policy as everything else:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE app.chunks (
  tenant_id   uuid NOT NULL REFERENCES app.tenants(id),
  id          bigint GENERATED ALWAYS AS IDENTITY,
  document_id uuid NOT NULL,
  content     text NOT NULL,
  embedding   vector(1024) NOT NULL,
  PRIMARY KEY (tenant_id, id),
  FOREIGN KEY (tenant_id, document_id) REFERENCES app.documents (tenant_id, id) ON DELETE CASCADE
);
ALTER TABLE app.chunks OWNER TO app_owner;
ALTER TABLE app.chunks ENABLE ROW LEVEL SECURITY;
ALTER TABLE app.chunks FORCE ROW LEVEL SECURITY;
CREATE POLICY tenant_isolation ON app.chunks
  USING      (tenant_id = (SELECT app.current_tenant()))
  WITH CHECK (tenant_id = (SELECT app.current_tenant()));
GRANT SELECT, INSERT, UPDATE, DELETE ON app.chunks TO app_user;

CREATE INDEX chunks_embedding_hnsw ON app.chunks USING hnsw (embedding vector_cosine_ops);
CREATE INDEX chunks_tenant ON app.chunks (tenant_id);

There is a performance trap. An HNSW index returns candidates first and filters afterward, so when a tenant owns a small share of the table, RLS filters out most candidates and the query returns too few results. pgvector 0.8.0 added iterative index scans for this: the scan continues until enough rows pass the filter or hnsw.max_scan_tuples (20,000 by default) is reached.

-- Inside the tenant transaction, before the similarity query
SET LOCAL hnsw.iterative_scan = relaxed_order;
SET LOCAL hnsw.ef_search = 100;

SELECT id, document_id, content, embedding <=> $1::vector AS distance
FROM app.chunks
ORDER BY embedding <=> $1::vector
LIMIT 8;

Partitioning the table by tenant_id, or partial indexes for your largest tenants, also help, with RLS kept as the safety net. With a dedicated vector database, use its native isolation primitive rather than a metadata filter. Pinecone's documentation recommends one namespace per tenant on serverless indexes, notes that namespaces are stored separately and query cost scales with namespace size, and that offboarding is deleting the namespace. A forgotten metadata filter queries every tenant; a namespace makes the tenant part of the address. Either way, enforce scoping in one retrieval function and never let feature code build raw vector queries. For a deeper look at retrieval design itself, see our guide to agentic RAG retrieval patterns that survive production.

In caches and prompts: the tenant is part of the key

Provider prompt caching caches the processed prefix of a prompt, not the response, and only hits on an identical prefix. Anthropic isolates caches between organizations (and, on the Claude API, between workspaces); OpenAI does not share caches across organizations. All your tenants share your organization, though, so put shared static content (instructions, tool definitions) first and tenant content after it.

Application response caching is the dangerous one. A semantic cache not keyed by tenant will return tenant A's answer, built from tenant A's documents, to a similar question from tenant B. Every cache key starts with the tenant id, and semantic caches are partitioned per tenant:

export function cacheKey(kind: string, ...parts: string[]): string {
  const { tenantId } = ctx();
  return ["t", tenantId, kind, ...parts].join(":");
}
// cacheKey("answer", promptVersion, sha256(normalizedQuestion))
// => "t:3f2c...:answer:v14:9b1e..."

Part 4: Authentication and Authorization

You need three kinds of identity: people who sign in, enterprises that manage those people centrally, and programs that call your API. Treat all three as principals acting for a tenant.

Organizations, memberships, and roles

Users belong to tenants through memberships, because the same person will eventually need two organizations. Roles live on the membership:

CREATE SCHEMA control AUTHORIZATION app_owner;
GRANT USAGE ON SCHEMA control TO app_user;

CREATE TYPE control.role AS ENUM ('owner', 'admin', 'member', 'viewer', 'billing');

CREATE TABLE control.memberships (
  tenant_id  uuid NOT NULL REFERENCES app.tenants(id),
  user_id    uuid NOT NULL,
  role       control.role NOT NULL DEFAULT 'member',
  created_at timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (tenant_id, user_id)
);

CREATE TABLE control.api_keys (
  id           uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  tenant_id    uuid NOT NULL REFERENCES app.tenants(id),
  prefix       text UNIQUE NOT NULL,       -- shown in the UI, used for lookup
  secret_hash  bytea NOT NULL,             -- sha256 of the full key
  scopes       text[] NOT NULL DEFAULT '{}',
  last_used_at timestamptz,
  expires_at   timestamptz,
  revoked_at   timestamptz
);

Keep authorization in one permission map from role to permissions rather than role checks scattered through handlers. AI features deserve their own permissions: using the assistant, running bulk AI jobs, and changing the AI budget are different powers.

SSO and SAML for enterprise tiers

Enterprise buyers expect single sign-on through their identity provider over SAML 2.0 or OpenID Connect, and often SCIM for automatic provisioning and deprovisioning. Configure SSO per tenant, keyed to a verified email domain; make enforcement a tenant setting that rejects password logins; have deprovisioning revoke sessions and the user's API keys; and map identity provider groups to membership roles. Most teams should buy SAML handling from an identity vendor, since hand-rolled signature validation has a long history of vulnerabilities, and own only the mapping from identity to membership. SSO is also a clear enterprise tier entitlement, next to AI limits.

API keys done properly

API keys often carry the highest spend potential, because scripts do not get tired. Generate keys with high entropy, show them once, store only a hash, and look them up by a non-secret prefix:

// api-keys.ts
import { createHash, randomBytes, timingSafeEqual } from "node:crypto";
import type { Pool } from "pg";

export function generateApiKey(env: "live" | "test") {
  const prefix = `sk_${env}_${randomBytes(6).toString("base64url")}`;
  const secret = randomBytes(32).toString("base64url");
  const fullKey = `${prefix}.${secret}`;
  return { prefix, fullKey, hash: createHash("sha256").update(fullKey).digest() };
}

export async function verifyApiKey(pool: Pool, presented: string) {
  const prefix = presented.split(".")[0];
  const { rows } = await pool.query(
    `SELECT id, tenant_id, secret_hash, scopes
       FROM control.api_keys
      WHERE prefix = $1 AND revoked_at IS NULL
        AND (expires_at IS NULL OR expires_at > now())`,
    [prefix],
  );
  const row = rows[0];
  if (!row) return null;
  const hash = createHash("sha256").update(presented).digest();
  if (hash.length !== row.secret_hash.length || !timingSafeEqual(hash, row.secret_hash)) return null;
  // Update last_used_at asynchronously or in batches to avoid a write per request.
  return { keyId: row.id as string, tenantId: row.tenant_id as string, scopes: row.scopes as string[] };
}

Plain SHA-256 is fine here, unlike for passwords, because the key is 32 random bytes. Give every key scopes, its own rate limit, and an optional budget, so a leaked key's damage is capped by configuration. Control-plane tables sit outside RLS because authentication reads them before a tenant is known, so restrict them with grants and route administrative changes through permission-checked functions.

Part 5: Usage Metering Design

Metering turns work into money. It must be accurate enough to invoice from, fast enough to enforce limits, and durable through retries and duplicate deliveries. The pattern is the one financial systems have used for decades: immutable events, idempotent writes, and an append-only ledger.

The usage event

Record one usage event per billable action, with raw token counts and cost, so you can recompute and reconcile later.

CREATE TABLE app.usage_events (
  tenant_id          uuid NOT NULL REFERENCES app.tenants(id),
  id                 bigint GENERATED ALWAYS AS IDENTITY,
  idempotency_key    text NOT NULL,
  occurred_at        timestamptz NOT NULL DEFAULT now(),
  feature            text NOT NULL,            -- 'assistant.chat', 'docs.summarize'
  actor_id           text NOT NULL,
  provider           text,                     -- 'anthropic', 'openai'
  model              text,
  input_tokens       integer NOT NULL DEFAULT 0,   -- uncached input
  cache_write_tokens integer NOT NULL DEFAULT 0,
  cache_read_tokens  integer NOT NULL DEFAULT 0,
  output_tokens      integer NOT NULL DEFAULT 0,
  cost_micros        bigint  NOT NULL DEFAULT 0,   -- your cost, in millionths of a dollar
  credits            numeric(18,4) NOT NULL DEFAULT 0, -- what the customer is charged
  prompt_version     text,
  request_id         text,
  billed_at          timestamptz,
  PRIMARY KEY (tenant_id, id),
  UNIQUE (tenant_id, idempotency_key)
);
ALTER TABLE app.usage_events OWNER TO app_owner;
ALTER TABLE app.usage_events ENABLE ROW LEVEL SECURITY;
ALTER TABLE app.usage_events FORCE ROW LEVEL SECURITY;
CREATE POLICY tenant_isolation ON app.usage_events
  USING      (tenant_id = (SELECT app.current_tenant()))
  WITH CHECK (tenant_id = (SELECT app.current_tenant()));
GRANT SELECT, INSERT ON app.usage_events TO app_user;

CREATE INDEX usage_unbilled ON app.usage_events (tenant_id, occurred_at) WHERE billed_at IS NULL;

Cost is integer micros, never floating point. The unique idempotency constraint means a retried request cannot record usage twice. And cost_micros (what you paid) is separate from credits (what you charge), because margin and allowances make them diverge.

Idempotency keys

Derive the key from the unit of work, not randomly at insert time: the client's Idempotency-Key header or request id plus step for requests, the job key plus step for jobs, the run id plus step index for agents. If the same work runs twice, it must produce the same key.

// metering.ts
import type { PoolClient } from "pg";
import { ctx } from "./context";

export type Usage = {
  idempotencyKey: string;
  feature: string;
  provider?: string;
  model?: string;
  inputTokens?: number;
  cacheWriteTokens?: number;
  cacheReadTokens?: number;
  outputTokens?: number;
  costMicros: number;
  credits: number;
  promptVersion?: string;
};

/** Must run inside withTenant(); returns false if this usage was already recorded. */
export async function recordUsage(db: PoolClient, u: Usage): Promise<boolean> {
  const c = ctx();
  const res = await db.query(
    `INSERT INTO app.usage_events
       (tenant_id, idempotency_key, feature, actor_id, provider, model,
        input_tokens, cache_write_tokens, cache_read_tokens, output_tokens,
        cost_micros, credits, prompt_version, request_id)
     VALUES (app.current_tenant(), $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13)
     ON CONFLICT (tenant_id, idempotency_key) DO NOTHING`,
    [
      u.idempotencyKey, u.feature, c.actorId, u.provider ?? null, u.model ?? null,
      u.inputTokens ?? 0, u.cacheWriteTokens ?? 0, u.cacheReadTokens ?? 0, u.outputTokens ?? 0,
      u.costMicros, u.credits, u.promptVersion ?? null, c.requestId,
    ],
  );
  return res.rowCount === 1;
}

Write the usage event in the same transaction as the result it pays for (the saved summary, the stored message). Both commit or neither does, which eliminates the most common metering bug.

The ledger pattern for credits and budgets

For prepaid credits or real-time budgets, do not keep a mutable balance column. Keep an append-only ledger of signed entries and derive the balance:

CREATE TYPE app.ledger_kind AS ENUM ('grant', 'purchase', 'reserve', 'release', 'debit', 'expire', 'adjust');

CREATE TABLE app.credit_ledger (
  tenant_id   uuid NOT NULL REFERENCES app.tenants(id),
  id          bigint GENERATED ALWAYS AS IDENTITY,
  entry_key   text NOT NULL,              -- idempotency for ledger writes
  kind        app.ledger_kind NOT NULL,
  amount      numeric(18,4) NOT NULL,     -- positive adds credits, negative removes
  reference   text,                       -- usage event key, invoice id, grant id
  expires_at  timestamptz,
  created_at  timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (tenant_id, id),
  UNIQUE (tenant_id, entry_key)
);
ALTER TABLE app.credit_ledger OWNER TO app_owner;
ALTER TABLE app.credit_ledger ENABLE ROW LEVEL SECURITY;
ALTER TABLE app.credit_ledger FORCE ROW LEVEL SECURITY;
CREATE POLICY tenant_isolation ON app.credit_ledger
  USING      (tenant_id = (SELECT app.current_tenant()))
  WITH CHECK (tenant_id = (SELECT app.current_tenant()));
GRANT SELECT, INSERT ON app.credit_ledger TO app_user;

CREATE VIEW app.credit_balance WITH (security_invoker = true) AS
SELECT tenant_id, coalesce(sum(amount), 0) AS balance
FROM app.credit_ledger
GROUP BY tenant_id;
GRANT SELECT ON app.credit_balance TO app_user;

The application role can insert but not update or delete, so corrections are new adjust entries. security_invoker keeps the ledger's RLS policy in force through the view. At scale, add periodic balance snapshots. Stripe's documentation describes its own credit grants as "backed by an immutable, append-only ledger"; yours is the one you can enforce against in real time.

Aggregation and pushing usage to billing

Billing systems should receive aggregated usage, not every token event. With Stripe's Meters API, the details that matter are:

• Events go to POST /v1/billing/meter_events with an event_name, a payload holding the customer id and numeric value, and optional identifier and timestamp.
• Stripe enforces identifier uniqueness "within a rolling period of at least 24 hours," so a deterministic identifier is your deduplication key.
• Timestamps must be within the past 35 days and at most 5 minutes ahead, and usage must arrive before the period's invoice finalizes.
• Live mode allows 1,000 meter event calls per second, one concurrent call per customer per meter; v2 meter event streams accept up to 10,000 events per second.
• Processing is asynchronous, so subscribe to v1.billing.meter.error_report_triggered. Cancellation via meter event adjustments works only within 24 hours.

Use a two-step outbox: claim unbilled events and write aggregated outbox rows in one statement, then send unsent rows using the outbox id as the identifier:

CREATE TABLE control.billing_outbox (
  id                 bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  tenant_id          uuid NOT NULL,
  stripe_customer_id text NOT NULL,
  event_name         text NOT NULL,
  value              numeric(18,4) NOT NULL,
  period_start       timestamptz NOT NULL,
  created_at         timestamptz NOT NULL DEFAULT now(),
  sent_at            timestamptz
);

// billing-sync.ts
import Stripe from "stripe";
import type { Pool } from "pg";

const stripe = new Stripe(process.env.STRIPE_SECRET_KEY!);

/** Step 1: claim closed usage and aggregate it into the outbox, atomically. */
export async function claimUsage(pool: Pool, graceMinutes = 10) {
  await pool.query(
    `WITH claimed AS (
       UPDATE app.usage_events SET billed_at = now()
        WHERE billed_at IS NULL
          AND occurred_at < date_trunc('hour', now() - make_interval(mins => $1))
       RETURNING tenant_id, credits, date_trunc('hour', occurred_at) AS hour
     )
     INSERT INTO control.billing_outbox (tenant_id, stripe_customer_id, event_name, value, period_start)
     SELECT c.tenant_id, b.stripe_customer_id, 'ai_credits', sum(c.credits), c.hour
       FROM claimed c
       JOIN control.billing_accounts b ON b.tenant_id = c.tenant_id
      GROUP BY c.tenant_id, b.stripe_customer_id, c.hour`,
    [graceMinutes],
  );
}

/** Step 2: send unsent rows. A crash after create() resends the same identifier. */
export async function sendOutbox(pool: Pool) {
  const { rows } = await pool.query<{
    id: string; stripe_customer_id: string; event_name: string; value: string; period_start: Date;
  }>(`SELECT id, stripe_customer_id, event_name, value::text, period_start
        FROM control.billing_outbox
       WHERE sent_at IS NULL
       ORDER BY id
       LIMIT 500`);

  for (const r of rows) {
    await stripe.billing.meterEvents.create({
      event_name: r.event_name,
      identifier: `outbox_${r.id}`,
      timestamp: Math.floor(r.period_start.getTime() / 1000),
      payload: { stripe_customer_id: r.stripe_customer_id, value: r.value },
    });
    await pool.query("UPDATE control.billing_outbox SET sent_at = now() WHERE id = $1", [r.id]);
  }
}

Each event is either unbilled or in exactly one outbox row. Late events land in a new row for the same hour, which Stripe sums. If the sender crashes after create(), the retry reuses outbox_<id> and Stripe discards the duplicate within its deduplication window, so alert on rows unsent for hours. These workers read across tenants, so they run as a dedicated billing role, never as the application role.

Reconcile against the provider

Daily, compare cost_micros per provider, model, and day with the provider's usage reports. Gaps point to calls that bypassed the gateway, stale price tables, or uncounted token fields (cache writes and thinking tokens are the usual suspects).

Part 6: Pricing Models for AI Features

Every pricing model implies different meters, limits, and enforcement points, so choose the pricing model first and design the events to support it.

Price in your units, not the provider's

Customers do not want to reason about tokens, and your price list should not move whenever a provider changes rates. Use an internal credit unit: each feature has a credit price per action (a chat answer is 1 credit, a summary 5, a research run 50), while the gateway tracks actual token cost for margin. You can then swap models behind a feature without changing what customers pay. Map every pricing component to a usage event before launch: a seat is a membership row, an AI action is a usage_events row with credits, an allowance is a recurring ledger grant, an overage is credits consumed beyond the allowance, and an outcome is a domain event with its own usage row.

What billing platforms handle, and what they do not

Billing platforms rate usage, invoice, and collect payment. They are not your real-time enforcement layer. From Stripe's documentation in October 2026:

• Meters aggregate events with sum, count, or last and attach to metered prices, processed asynchronously.
• Billing credits (credit grants) apply only to metered prices that report through Meters, and only at invoice finalization. They do not stop a request at zero balance; your ledger does.
• Advanced usage-based billing (pricing plans, rate cards, recurring credit grants) runs on the v2 billing API with a preview API version in the docs reviewed, so check its status before building on it.
• Metronome, acquired by Stripe in January 2026, covers prepaid commits with burn-down, auto-recharge, enterprise commits, and dimensional pricing. Stripe's "Bill for LLM tokens," which syncs OpenAI, Anthropic, and Google token prices and applies your markup, runs through Metronome and is in public preview.

The architecture is the same either way: your application owns the real-time balance and hard limits; the billing platform owns invoices and payment.

Part 7: Model Cost Controls

Model cost is a formula with a handful of inputs, and every control in this section moves one of them.

The cost formula

For a single model call, cost is the sum of four token buckets, each at its own price:

cost = (U × P_in + W × P_write + R × P_read + O × P_out) / 1,000,000

where U is uncached input tokens, W is tokens written to the prompt cache, R is tokens read from the cache, O is output tokens (including any thinking tokens), and each P is the provider's price per million tokens. Anthropic's documentation spells out the relationship directly: total input tokens equal cache_read_input_tokens + cache_creation_input_tokens + input_tokens, and the input_tokens field only counts tokens after the last cache breakpoint.

Anthropic list prices from its documentation in October 2026 (load them from configuration, not constants):

Haiku 5.5 prompts over 100,000 tokens cost $0.50 input and $2.50 output per million. The Message Batches API halves input and output prices. OpenAI has a similar shape (default prompt caching, cached tokens in input_tokens_details.cached_tokens, a 50 percent Batch API discount), but cache rules vary by model generation.

A worked example

Consider a hypothetical support assistant (illustrative numbers, not customer data). Each request sends a shared prefix of 6,000 tokens (system instructions, policies, and tool definitions), plus 4,000 tokens of retrieved context, conversation history, and the user's question, and receives about 700 output tokens.

Without caching, on Sonnet 5.5: 10,000 input tokens at $2.00 is $0.0200, plus 700 output tokens at $10.00 is $0.0070, for $0.0270 per request.

With prompt caching on the shared prefix, assume a 95 percent cache hit rate. The prefix costs 6,000 × (0.95 × $0.10 + 0.05 × $2.50) / 1,000,000 = $0.00132. The uncached 4,000 tokens cost $0.0080 and output still costs $0.0070, for $0.01632 per request, about 40 percent less.

With routing, a small classifier on Haiku 5.5 (about 800 input and 10 output tokens, $0.000085) sends the 60 percent of questions that are simple lookups to Haiku 5.5, where the same request shape costs about $0.00084. The blended cost is 0.6 × $0.00084 + 0.4 × $0.01632 + $0.000085, or roughly $0.0071 per request.

Now scale it to one tenant with 50 seats, each sending 30 requests per working day for 22 days: 33,000 requests a month.

Cache-friendly prompts and routing moved model cost from nearly half of revenue to about a tenth with no product change. It also shows the risk of seat pricing alone: a seat sending 300 requests a day costs ten times as much, and nothing in a seat price absorbs that. For offline work, a nightly job that summarizes 2,000 documents of 8,000 input tokens each, with 500 output tokens per summary, costs about $42 per run on Sonnet 5.5 at standard rates and about $21 through the Batch API.

Per-tenant budgets with reserve-then-settle

A budget check after the call is a report, not a control. Reserve the worst-case cost before the call and settle with the actual cost after. The worst case is bounded because you set max_tokens and can count input tokens first.

// budget.ts
import type { PoolClient } from "pg";

export class BudgetExceeded extends Error {}

/** Reserves credits inside the tenant transaction. Serialized per tenant by an advisory lock. */
export async function reserve(db: PoolClient, key: string, credits: number) {
  await db.query("SELECT pg_advisory_xact_lock(hashtextextended(app.current_tenant()::text, 0))");
  const { rows } = await db.query<{ balance: string }>(
    "SELECT coalesce(sum(amount), 0)::text AS balance FROM app.credit_ledger",
  );
  if (Number(rows[0].balance) < credits) throw new BudgetExceeded("Insufficient AI credits");
  await db.query(
    `INSERT INTO app.credit_ledger (tenant_id, entry_key, kind, amount, reference)
     VALUES (app.current_tenant(), $1, 'reserve', $2, $3)
     ON CONFLICT (tenant_id, entry_key) DO NOTHING`,
    [`reserve:${key}`, -credits, key],
  );
}

/** Releases the reservation and debits the actual amount, idempotently. */
export async function settle(db: PoolClient, key: string, reserved: number, actual: number) {
  await db.query(
    `INSERT INTO app.credit_ledger (tenant_id, entry_key, kind, amount, reference) VALUES
       (app.current_tenant(), $1, 'release', $2, $4),
       (app.current_tenant(), $3, 'debit',   $5, $4)
     ON CONFLICT (tenant_id, entry_key) DO NOTHING`,
    [`release:${key}`, reserved, `debit:${key}`, key, -actual],
  );
}

The reservation commits in its own short transaction, so no lock is held during the call. A sweeper releases reservations older than your maximum call duration, and idempotent entry keys make retries safe.

Rate limits and token caps

Budgets cap spend per month; rate limits cap it per minute, protecting you from runaway loops and from one tenant starving others of provider quota. Anthropic rate-limits requests, input tokens, and output tokens per minute with a token bucket, so mirror that per tenant and feature:

// rate-limit.ts
import Redis from "ioredis";
const redis = new Redis(process.env.REDIS_URL!);

// KEYS[1] bucket key; ARGV: capacity, refill per second, cost, now in ms
const TOKEN_BUCKET = `
local cap = tonumber(ARGV[1]); local rate = tonumber(ARGV[2])
local cost = tonumber(ARGV[3]); local now = tonumber(ARGV[4])
local b = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local tokens = tonumber(b[1]) or cap; local ts = tonumber(b[2]) or now
tokens = math.min(cap, tokens + (now - ts) / 1000 * rate)
local ok = 0
if tokens >= cost then tokens = tokens - cost; ok = 1 end
redis.call('HSET', KEYS[1], 'tokens', tokens, 'ts', now)
redis.call('PEXPIRE', KEYS[1], math.ceil(cap / rate * 1000) + 1000)
return ok`;

/** Returns true if the tenant may spend `cost` tokens now. */
export async function allow(tenantId: string, feature: string, cost: number, perMinute: number) {
  const ok = await redis.eval(
    TOKEN_BUCKET, 1, `rl:${tenantId}:${feature}`,
    perMinute, perMinute / 60, cost, Date.now(),
  );
  return ok === 1;
}

Charge the bucket with estimated input plus max_tokens. Then cap tokens: set max_tokens per feature, summarize history beyond a fixed window, and limit retrieved chunks. Output includes thinking tokens, so effort settings (Anthropic's output_config.effort) are cost controls too.

Model routing by task

Most products have a few task types (classify, extract, summarize, answer, act), and they do not all need your most capable model. Make routing configuration: a table keyed by task and plan returns a model, a max_tokens cap, and a batchable flag: for example, classification and extraction on Haiku 5.5, summaries and answers on Sonnet 5.5, and agent runs on Opus 5.5 for enterprise plans. Change a route only when the cheaper model passes a labeled evaluation set for that task.

Caching, batching, and kill switches

Prompt caching is usually the largest lever. Put stable content first and the breakpoint after it. Anthropic uses cache_control (up to four breakpoints, 5-minute or 1-hour lifetime), reports hits in cache_read_input_tokens, and silently skips prompts under the model minimum (512 tokens on the Claude 5.5 models). OpenAI enables prompt caching by default (with a 1,024-token minimum on its newest models) and accepts a prompt_cache_key, which helps route related requests to the same cache on older models and, on newer ones, separates cache accounting per customer or workspace.

Batch processing suits anything that need not answer in seconds: summaries, evaluations, bulk classification, backfills. Anthropic's Message Batches API takes up to 100,000 requests or 256 MB per batch, finishes most within an hour, expires work not done in 24 hours, and keeps results 29 days.

Kill switches, checked on every call and cached for seconds, should exist per provider, per feature, and per tenant, so a runaway loop or injection campaign stops without a deploy.

Putting it together in the gateway

// gateway.ts
import Anthropic from "@anthropic-ai/sdk";
import { randomUUID } from "node:crypto";
import { ctx } from "./context";
import { withTenant } from "./tenant-db";
import { reserve, settle } from "./budget";
import { allow } from "./rate-limit";
import { recordUsage } from "./metering";

const anthropic = new Anthropic({ timeout: 60_000, maxRetries: 2 });

type Price = { in: number; write5m: number; read: number; out: number; creditsPerUsd: number };
declare function priceFor(model: string): Price;                 // from config
declare function killSwitchOn(feature: string, tenantId: string): Promise<boolean>;
declare function route(task: string, plan: string): { model: string; maxTokens: number };

export async function generate(opts: {
  feature: string;
  task: "classify" | "extract" | "summarize" | "answer" | "agent";
  system: string;                    // stable, cacheable prefix
  messages: Anthropic.MessageParam[];
  promptVersion: string;
  idempotencyKey?: string;
}) {
  const c = ctx();
  if (await killSwitchOn(opts.feature, c.tenantId)) throw new Error("Feature temporarily disabled");

  const r = route(opts.task, c.plan);
  const p = priceFor(r.model);
  const key = opts.idempotencyKey ?? `${c.requestId}:${opts.feature}:${randomUUID()}`;

  const system: Anthropic.TextBlockParam[] = [
    { type: "text", text: opts.system, cache_control: { type: "ephemeral" } },
  ];
  const { input_tokens: estIn } = await anthropic.messages.countTokens({
    model: r.model, system, messages: opts.messages,
  });
  if (!(await allow(c.tenantId, opts.feature, estIn + r.maxTokens, 400_000))) {
    throw new Error("Rate limit exceeded for this workspace");
  }

  // Worst case: every input token uncached and written to cache, full max_tokens of output.
  const worstUsd = (estIn * Math.max(p.in, p.write5m) + r.maxTokens * p.out) / 1e6;
  const reserved = worstUsd * p.creditsPerUsd;
  await withTenant(c.tenantId, (db) => reserve(db, key, reserved));

  try {
    const msg = await anthropic.messages.create(
      { model: r.model, max_tokens: r.maxTokens, system, messages: opts.messages },
      { timeout: 45_000 },
    );
    const u = msg.usage;
    const write = u.cache_creation_input_tokens ?? 0;
    const read = u.cache_read_input_tokens ?? 0;
    const usd = (u.input_tokens * p.in + write * p.write5m + read * p.read + u.output_tokens * p.out) / 1e6;
    const credits = usd * p.creditsPerUsd;

    await withTenant(c.tenantId, async (db) => {
      await settle(db, key, reserved, credits);
      await recordUsage(db, {
        idempotencyKey: key, feature: opts.feature, provider: "anthropic", model: r.model,
        inputTokens: u.input_tokens, cacheWriteTokens: write, cacheReadTokens: read,
        outputTokens: u.output_tokens, costMicros: Math.round(usd * 1e6), credits,
        promptVersion: opts.promptVersion,
      });
    });
    return msg;
  } catch (err) {
    await withTenant(c.tenantId, (db) => settle(db, key, reserved, 0));
    throw err;
  }
}

For brevity, credits here derive from cost; most products charge a fixed credit price per action and keep cost for margin tracking. Either way, every call passes the same gates in order and ends with a settled ledger entry and a usage event.

Part 8: Observability

The usage events from Part 5 contain most of what you need; traces and prompt versions add the rest.

Per-tenant cost dashboards

You should be able to answer which tenants and features cost the most, and which tenants are unprofitable, with plain SQL:

-- Model cost and gross margin per tenant, current month
WITH cost AS (
  SELECT tenant_id,
         sum(cost_micros) / 1e6                         AS model_cost_usd,
         sum(cache_read_tokens)::numeric
           / nullif(sum(input_tokens + cache_write_tokens + cache_read_tokens), 0) AS cache_hit_ratio
  FROM app.usage_events
  WHERE occurred_at >= date_trunc('month', now())
  GROUP BY tenant_id
)
SELECT t.name, r.mrr_usd, c.model_cost_usd,
       round(c.cache_hit_ratio, 3)                                AS cache_hit_ratio,
       round(1 - c.model_cost_usd / nullif(r.mrr_usd, 0), 3)      AS ai_gross_margin
FROM cost c
JOIN app.tenants t ON t.id = c.tenant_id
LEFT JOIN control.tenant_revenue r ON r.tenant_id = c.tenant_id
ORDER BY c.model_cost_usd DESC
LIMIT 50;

Run it as an analytics role. Alert when a tenant's daily cost exceeds three times its seven-day average, or when a feature's cache hit ratio drops after a deploy.

Traces with tenant and cost attributes

Traces show where time and money go inside an agent run. OpenTelemetry's generative AI semantic conventions, described by the project as under active development, define attributes such as gen_ai.request.model, gen_ai.usage.input_tokens, and gen_ai.usage.output_tokens. Add your own: app.tenant_id, app.feature, app.prompt_version, and app.cost_micros on every model span. Keep tenant content (prompts, documents, answers) out of span attributes by default, because traces often flow to third-party tools with broader access.

Prompt and version logging

Version every prompt template in a registry and record the version on every usage event, so "what changed?" is a query. Log full prompt content only where needed, with redaction, per-tenant retention, and an opt-out.

Part 9: Reliability

Timeouts

Set explicit timeouts per task: interactive features fail fast, batch jobs wait. Anthropic recommends streaming or batches for long requests, and its SDKs reject non-streaming requests expected to exceed 10 minutes. With streaming, add an idle timeout, and handle errors that arrive mid-stream after a 200.

Retries, done carefully

In Anthropic's error reference, 429, 500, and 529 (overloaded_error) are transient, and the SDKs retry them twice by default, honoring retry-after. But a spend-cap 429 has no retry-after and keeps failing, and a workspace spend limit returns a 400. Beware retry multiplication: SDK, gateway, and queue retries stack, so one failing call becomes dozens. Give each failure type one owning layer and keep every retried unit idempotent.

Fallbacks

Fallbacks keep features working during overloads: the same model on a second platform (Claude on Amazon Bedrock or Google Cloud), a smaller model, or another provider. Evaluate fallback routes like primary ones, expect higher cost because caches do not transfer, and pass fallbacks through the same gateway gates.

Queues and fairness

Anything that need not answer in seconds belongs on a queue, which absorbs rate limits and enables batch APIs. Add per-tenant worker concurrency limits so one tenant's 50,000-document upload does not delay everyone, weight by plan if you like, and alert on dead-letter depth.

Circuit breakers

A circuit breaker stops traffic to a failing model briefly and routes to the fallback. It handles automatic protection; kill switches handle human decisions.

Part 10: Security and Compliance Basics

AI adds one new attack surface to SaaS security: the model, which reads untrusted text and can be persuaded to act on it.

Data isolation, end to end

Write down an explicit isolation model: every store that holds tenant data and the mechanism that isolates it. Enterprise security reviews ask for exactly this, and the exercise usually finds a store nobody considered, such as an analytics export, a log pipeline, or an evaluation dataset.

Encryption

Use TLS in transit and managed encryption at rest. For enterprise tiers, consider per-tenant keys with envelope encryption in a key management service (AWS KMS or its Google Cloud and Azure equivalents). Deleting a tenant's key makes their encrypted backups unreadable, and customer-managed keys become possible later without a storage redesign.

Retention

Prompts, completions, histories, embeddings, evaluation sets, and traces each need a retention period, configurable per tenant for enterprise plans, and deletion must reach vector indexes and caches. Embeddings are derived from customer content, so treat them as customer data. Both Anthropic and OpenAI document zero data retention arrangements for eligible customers; confirm the features you rely on work under them.

Audit logs

Keep an append-only log of security-relevant actions per tenant: sign-ins, SSO and role changes, API keys, budget changes, exports, and AI actions that change data or call external systems:

CREATE TABLE app.audit_log (
  tenant_id   uuid NOT NULL REFERENCES app.tenants(id),
  id          bigint GENERATED ALWAYS AS IDENTITY,
  occurred_at timestamptz NOT NULL DEFAULT now(),
  actor_type  text NOT NULL,         -- 'user', 'api_key', 'system', 'ai_agent'
  actor_id    text NOT NULL,
  action      text NOT NULL,         -- 'api_key.created', 'budget.updated', 'agent.tool_call'
  target      text,
  metadata    jsonb NOT NULL DEFAULT '{}',
  request_id  text,
  PRIMARY KEY (tenant_id, id)
);
ALTER TABLE app.audit_log OWNER TO app_owner;
ALTER TABLE app.audit_log ENABLE ROW LEVEL SECURITY;
ALTER TABLE app.audit_log FORCE ROW LEVEL SECURITY;
CREATE POLICY tenant_isolation ON app.audit_log
  USING      (tenant_id = (SELECT app.current_tenant()))
  WITH CHECK (tenant_id = (SELECT app.current_tenant()));
GRANT SELECT, INSERT ON app.audit_log TO app_user;

Expose it to tenant admins; it is one of the cheapest, most requested enterprise features.

Prompt injection boundaries between tenants

Prompt injection is first on OWASP's 2025 list, and no model is fully immune, so limit what a successful injection can reach:

1. One model context, one tenant. Never mix two tenants' content in a prompt, memory, or agent run.
2. Tools run with the caller's permissions. Tools execute in the same tenant context and user role as the run, and never accept a tenant id from model output. If the model can name the tenant, an injected document can too.
3. Retrieved content is data, not instructions. It must never change which tools are available or which tenant is in scope.
4. High-impact actions need confirmation outside the model: sending email, spending money, exporting data.
5. Validate outputs against schemas, escape before rendering, and sandbox any generated code.
6. Keep secrets out of prompts. OWASP lists system prompt leakage separately as LLM07.

These boundaries matter more when agents reach external systems through MCP; our explainer on MCP as the integration layer behind production AI agents covers how to scope tool access per user and per tenant.

Unbounded consumption is a security issue too

OWASP's LLM10, "Unbounded Consumption," makes every Part 7 control a security control. A leaked key without a rate limit and budget is a direct line to your model spend.

Part 11: Failure Patterns

These show up repeatedly once multi-tenant AI products meet real usage.

The application connects as the table owner or an admin role. Every policy is silently bypassed. Check current_user and its attributes from inside the running application.

Tenant context set per session behind a transaction pooler. A plain SET persists on the connection and the next tenant inherits it. Use set_config(..., true).

Jobs without a tenant. Workers write data or spend money outside any tenant. Require and validate the tenant in every envelope.

A semantic cache shared across tenants. Keyed by the normalized question, it serves tenant A's answer to tenant B.

Vector search filtered after the fact. One code path forgets the filter. Make the tenant part of the query's address.

Usage recorded outside the transaction. Metered usage drifts from delivered work. Write both in one transaction.

Budgets checked after the call. An agent loop blows through the budget overnight. Reserve before, settle after.

Retry multiplication. Stacked retries turn an incident into a traffic spike and duplicate charges.

Dynamic content in front of the cache breakpoint. A timestamp at the top of the system prompt drops the cache hit ratio to near zero and costs jump with flat traffic.

Unlimited AI on a seat price. A few heavy users or scripts push model cost past their tenant's revenue. Every plan needs an allowance and a ceiling.

Pricing in raw tokens. Model upgrades change tokenization: Anthropic notes Claude Haiku 5.5 counts the same text as roughly 30 percent more tokens than Haiku 4.5. Price in your own credits.

No per-tenant cost visibility. Nobody can say which customers are profitable. Store cost per event from day one.

Part 12: A Prioritized Build Plan

This order reduces risk fastest, whether you are building new or retrofitting.

1. Tenant model and RLS (weeks 1 to 2). Tenant-first keys, forced RLS, separate owner and application roles, the CI check, and the routing function.
2. Tenant context everywhere (week 2). Request context, withTenant transactions, job envelopes, tenant-prefixed caches, scoped retrieval.
3. The AI gateway (weeks 2 to 3). One path for every model call, with routing, timeouts, token caps, caching, and token accounting. Lint against direct SDK imports.
4. Usage events and the ledger (weeks 3 to 4). Idempotent usage events in the result's transaction, plus the append-only ledger.
5. Budgets, rate limits, and kill switches (week 4). Reserve-then-settle budgets, token-bucket limits, and kill switches at three levels.
6. Per-tenant cost visibility (weeks 4 to 5). Cost and margin by tenant, feature, and model, cache hit ratio, anomaly alerts, daily reconciliation.
7. Billing integration (weeks 5 to 6). Map pricing to meters, build the outbox sync, subscribe to error events, and test invoice previews on real usage.
8. Auth for teams and enterprise (weeks 6 to 8). Memberships, AI permissions, scoped API keys with limits, and SSO and SCIM for enterprise plans.
9. Reliability hardening (weeks 8 to 9). Retry ownership, evaluated fallbacks, per-tenant queue concurrency, circuit breakers, batch APIs.
10. Security and compliance artifacts (ongoing). Isolation model, in-product audit log, retention, key strategy, and injection boundaries for every agent tool.

The weeks assume a small, experienced team and are a planning guide; the order matters more than the dates.

Conclusion: Make Cost and Isolation First-Class

The difference between an AI feature and an AI product is mostly invisible to users: tenant context on every request, retrieval that cannot return another customer's data, usage written with the work, and a gateway that knows each call's cost before it is made. None of these patterns is new. What is new is that AI makes them mandatory from the first paying customer.

Start with isolation and metering. Add the gateway before you add the second AI feature. Price in your own units. Then let the dashboards tell you where to optimize next, whether that is a cache breakpoint, a routing rule, or a plan that needs a ceiling.

Need help designing or hardening a multi-tenant AI product? Brightter, Inc. designs and builds custom web applications and AI features, from tenant isolation and metering to model cost controls and enterprise readiness. As an official Anthropic Claude Partner and Webflow Certified Partner, we work across the stack, from production AI integrations to the marketing site that sells them. Talk to our team about an architecture review or a build.