OpenStack Keystone in Rust

The Python Keystone identity service, maintained upstream by the OpenStack community, has served the OpenStack ecosystem reliably for years. It handles authentication, authorization, token issuance, service catalog, project/tenant management, and federation services across thousands of deployments. However, as we embarked on adding next-generation identity features—such as native WebAuthn (“passkeys”), modern federation flows, direct OIDC support, JWT login, workload authorization, restricted tokens and service-accounts—it became clear that certain design and performance limitations of the Python codebase would hamper efficient implementation of these new features.

Consequently, we initiated a project termed “Keystone-NG”: a Rust-based component that augments rather than fully replaces the existing Keystone service. The original plan was to implement only the new feature-set in Rust and route those new API paths to the Rust component, while keeping the core Python Keystone service in place for existing users and workflows.

As development progressed, however, the breadth of new functionality (and the opportunity to revisit some of the existing limitations) led to a partial re-implementation of certain core identity flows in Rust. This allows us to benefit from Rust’s memory safety, concurrency model, performance, and modern tooling, while still preserving the upstream Keystone Python service as the canonical “master” identity service, routing only the new endpoints and capabilities through the Rust component.

In practice, this architecture means:

• The upstream Python Keystone remains the main identity interface, preserving backward compatibility, integration with other OpenStack services, existing user workflows, catalogs, policies and plugins.

• The Rust “Keystone-NG” component handles new functionality, specifically:

  • Native WebAuthN (passkeys) support for passwordless / phishing-resistant MFA

  • A reworked federation service, enabling modern identity brokering and advanced federation semantics OIDC (OpenID Connect) Direct in Keystone, enabling Keystone to act as an OIDC Provider or integrate with external OIDC identity providers natively JWT login flows, enabling stateless, compact tokens suitable for new micro-services, CLI, SDK, and workload-to-workload scenarios

  • Workload Authorization, designed for service-to-service authorization in cloud native contexts (not just human users)

  • Restricted Tokens and Service Accounts, which allow fine-grained, limited‐scope credentials for automation, agents, and service accounts, with explicit constraints and expiry

By routing only the new flows through the Rust component we preserve the stability and ecosystem compatibility of Keystone, while enabling a forward-looking identity architecture. Over time, additional identity flows may be migrated or refactored into the Rust component as needed, but our current objective is to retain the existing Keystone Python implementation as the trusted, mature baseline and incrementally build the “Keystone-NG” Rust service as the complement.

We believe this approach allows the best of both worlds: the trusted maturity of Keystone’s Python code-base, combined with the modern, high-safety, high-performance capabilities of Rust where they matter most.

Compatibility

Highest priority is to ensure that this implementation is compatible with the original python Keystone: authentication issued by Rust implementation is accepted by the Python Keystone and vice versa. At the same time it is expected, that the new implementation may implement new features not supported by the Python implementation. In this case, it is still expected that such features do not break authentication flows. It must be possible to deploy Python and Rust implementation in parallel and do request routing on the web server level.

Database

Adding new features most certainly require having database changes. It is not expected that such changes interfere with the Python implementation to ensure it is working correctly.

API

Also here it is expected that new API resources are going to be added. As above it is not expected that such changes interfere with the Python implementation to ensure it is still working correctly and existing clients will not break.

Executive summary: the authentication and authorization vision

Status: Draft for community discussion

Date: 2026-10-15

This page is for readers who decide rather than implement: Technical Committee members, project leads and operators. It is one page by design. The vision holds the reasoning, the roadmap holds the order of work, and IAM data plane and check API holds the design of the part that carries the most risk.

The problem

Keystone predates the identity standards the rest of the industry is built on. Its token model, API and scope-and-roles payload were designed in 2010–2012, before OAuth 2.0 (RFC 6749, October 2012), OpenID Connect (2014), JWT (RFC 7519, 2015), SPIFFE (2017) and OAuth token exchange (RFC 8693, 2020). Those standards were attached at the edges later and never adopted as the core. Federation goes through Apache modules that map external assertions onto local users and groups. OAuth 1.0a and OAuth 2.0 client credentials are extensions with a limited coverage. JWS is just another token format carrying the same payload. X-Auth-Token and /v3/auth/tokens are still OpenStack-only, so no gateway, identity provider or service mesh can validate an OpenStack token or read its roles without calling Keystone. Each problem below has to be fixed inside Keystone instead of by adopting what other platforms already use.

An OpenStack token proves who the caller is, carries the project, domain or system the caller may act on, and carries the roles held there, all frozen at issuance. Every service validates it against Keystone and applies local policy to what is inside (§2). Following consequences hurt.

1. Every request is a Keystone round trip. A Keystone outage stops the cloud the moment the validation cache expires (§2.2).

2. Scope and roles in the token cannot express what operators need. Policy asks “does this token’s scope match the resource”, not “may this principal do this to that resource”. The two diverge in cross-project operations, cloud-wide read and services acting for users, so system scope, global reader and “admin is global” never worked cleanly and Secure RBAC stalled (§2.1, §2.3, §2.4).

3. Roles are the only unit of grant. To let one workload call one operation, an operator would create a role, wire it into every service’s policy file and assign it. Not all operators do this, and this is not triviel, so the workload gets member or admin. Application credential access rules can only narrow a credential that is already as privileged as its role, never grant (§2.5).

4. Long-lived secrets are everywhere (§2.6). Each is a bearer secret whose theft looks exactly like legitimate use.

• Every service has a service user with a password in a config file and the admin or service role.
• Heat, Magnum and Mistral store trusts. Magnum ships a trustee password into every cluster VM. Heat ships a per-instance EC2 keypair into the metadata of every VM that runs its CloudFormation client.
• Tenant automation runs on application credentials or EC2 keys baked into images, Kubernetes secrets and CI variables, valid for months.
• Users authenticate with passwords in clouds.yaml.

5. Trusts, application credentials and EC2 credentials are a recurring source of vulnerabilities. They are delegation mechanisms, and delegation is where Keystone has repeatedly decided on the wrong input: the scope of the token instead of the immutable record of how the caller authenticated (security model).

6. Keystone doesn’t scale well. A massive swift/rgw cluster requires scaling Keystone to hundreds+ of instances only to validate request signatures.

• OSSA-2026-005 / CVE-2026-33551. A restricted application credential escaped its role restriction by minting an EC2 credential. Redeeming it at /v3/ec2tokens rebuilt the delegation chain but presented a bare project scope, so the credential inherited the owner’s full project roles.
• OSSA-2026-015. A delegated token was not bound to its delegation project on credential create, read, update and delete, so a delegate could reach credentials outside the project it was delegated.
• Trusts have structural weaknesses besides bugs (ADR 0036). The trustee is a user with a password that is copied to wherever the service runs. expires_at is optional, so a trust need not lapse. Any user can name any other user as trustee, and nothing caps the roles a service can receive. With impersonation downstream logs see only the trustor. One trustee holds every trust of its service, so activity cannot be attributed per trust. OS-TRUST is a home-grown protocol that no gateway, identity provider or audit tool understands, and it is expressed as token scope, which blocks scope-less tokens.
• The pattern is the point. Fixing each advisory closes one path. The vision removes the scope-keyed decision itself, and removes the delegated secret, so the next variant has less to key on.

OpenStack is the outlier on credential, scope location and workload identity against AWS, GCP, Azure and Kubernetes (§2.7).

The proposal

Modernize keystone and separate the three things the token conflates (§1).

Four further changes follow:

• A secretless control plane. Services authenticate by SPIFFE identity. Service users, service passwords and X-Service-Token go away (§4.2).
• Trusts are replaced by server-side service grants and RFC 8693 on-behalf-of exchange. No secret is delegated, and every delegation is recorded (§4.5).
• Tenant workloads get identity without credentials. A VM or cluster node attests and receives an SVID bound to its project (§4.6).
• Roles stay, permissions are added. A rule, an operation or a set of them can be bound to any human, service or workload on any target, without inventing a role (§6).

What it costs

The costs fall on different parts of the community.

• Operators deploy and maintain SPIRE. The secretless control plane and tenant workload identity both need a SPIRE deployment: servers, an agent on every control-plane host and compute host, a trust domain, and the plumbing that registers each service. It is new infrastructure with its own upgrades, monitoring, backup and certificate-rotation practice, and it sits under every service-to-service call. Deployment tools and devstack have to ship it, and the roadmap makes this the first milestone (§4.2, A0).
• Operators also take on a new availability model. Authorization becomes a live check on every request, and an optional replicated data plane per failure domain keeps that from being a single point of failure. A total outage of the write path still stops all new logins, refreshes and exchanges, so the access-token lifetime becomes an availability setting (§8.5).
• Service maintainers change their policy and their API handling. A service asks instead of reading roles out of the token, and it enforces that the requested target matches the resource’s owner before it accepts per-request targets (§5.1). Each service may do this on its own schedule, or not at all.
• Client and SDK authors take on new key management. Public clients need sender-constrained tokens, and official OpenStack clients have no established story for holding a DPoP key yet (§10).
• The community has to agree on shared contracts. The header grammar and the operation vocabulary only work if projects adopt the same one, and who may write to the vocabulary is a decision about who holds a privilege (§6.1).

How it is adopted

Every step is additive (§9). Python Keystone, Fernet tokens and unmodified services keep working. Every capability has a per-service off switch. A deployment may stop at role resolution indefinitely and lose nothing it had.

The roadmap orders the work by dependency, not as one line of phases, so teams can start in parallel: SPIRE and the secretless control plane, the token format, the check API, and the policy engine.

Decisions needed

Most questions belong to the community, not to this document (§10): the OpenStack-Target header grammar, where the operation registry lives and who may write to it, DPoP versus mTLS-bound tokens for public clients, and the retirement timeline for application credentials and EC2 keys.

OpenStack Authentication and Authorization: A Vision

Status: Draft for community discussion

Date: 2026-09-18

This document proposes a target model for authentication and authorization across OpenStack. It is derived from the architecture decisions already taken in the Rust Keystone implementation (the ADRs under adr/) and from the SPIRE integration plan, and it extends them into one coherent picture that other OpenStack projects can react to. It is deliberately written for a wide audience: operators, service maintainers, SDK authors, and the Technical Committee. It is not an ADR and it does not commit any project to anything; its purpose is to give the community a concrete proposal to argue about.

The model is described independently of any implementation. The Rust Keystone is the reference implementation and the place where the building blocks already exist, but nothing in the proposal requires it; Python Keystone, keystonemiddleware, oslo.policy and every service can adopt the pieces that concern them.

Each part states its own requirements and limits where it is described, and the last section lists the questions the community, not this document, has to answer.

A one-page version for decision-makers is the executive summary.

This document says what the model is and why. The order in which it is adopted is in the roadmap. The design detail of the check API and the data plane is in IAM data plane and check API.

TL;DR

The problem: scope and roles are bound into the token at issuance, so one artifact proves who the caller is, where it may act and what it may do. The proposal separates the three.

• Tokens say who, and nothing else. Short-lived, scope-less JWTs from an OAuth2/OIDC provider. Public clients use DPoP (RFC 9449) and SVID holders use JWT-SVID client authentication (RFC 7523). Authentication is validated offline. Authorization is not: it is decided per request. During the transition scoped tokens remain available. A token still carries authentication facts such as amr, act, delegation_context and optional restrictions, and can be narrowed to the audience of one service type.
• The request says where. The target travels per request in the OpenStack-Target header, which generalizes the tokenless-auth headers of v3 Keystone, or is derived from the addressed resource. For an operation on an existing resource the resource’s owner wins over anything the caller names. For create and list the caller names the target, and Keystone checks the caller’s authority there.
• Keystone decides what. The service’s policy asks Keystone (/v4/authz/check) on every request, against a replica of live state, with no cross-request cache. An outage of the write path degrades the cloud rather than stopping it, to a bound the document states (§8.5).
• The control plane authenticates with SPIFFE mTLS. Services are identified by their SVIDs, not by passwords. Service users, service passwords and X-Service-Token go away. Each service is still a principal in Keystone, mapped from its SPIFFE ID, and holds the service permission set on the system target. mTLS identifies the service, never the user, so a call made for a user still carries the user’s JWT, forwarded or exchanged. A callee can also key a rule on the calling service’s identity without reading roles.
• Trusts are replaced. Service grants are server-side, revocable records for authority that must work with no user present. RFC 8693 on-behalf-of exchange continues a user’s operation. No secret is delegated, and the act and delegation_context claims record the chain (ADR 0036).
• Tenant workloads get identity without credentials. A VM or cluster node attests with a Keystone-signed proof of its placement and receives an SVID bound to its project. A mapping rule then grants it roles or permissions on that project. Application credentials and EC2 keys remain for off-cloud automation, as a legacy niche.
• Global reader and admin isolation become expressible. Because the target is named per request, a system-level grant such as reader can apply to project-owned resources without a project-less system-scoped token. Keystone resolves that projection in one place. Each service must still enforce that the requested target matches the resource’s owner, and a service adopts the model only after it makes that declaration.
• Separate interfaces. A public API; an internal interface that accepts SPIFFE mTLS (the plaintext back-channel stays only until legacy services have moved); and a local admin interface over a Unix domain socket that carries the full API, is reachable only from the host and authenticates by SVID.
• A secretless control plane. No Keystone credentials in the control plane and no long-lived inter-service secrets. Database and message-queue credentials are outside this proposal.

The order of adoption is in the roadmap.

1. The vision in one page

OpenStack authentication and authorization today is built around one artifact: the scoped bearer token. A token proves who the caller is and carries the project, domain or system the caller may act on and the roles the caller holds there, frozen at issuance. Every service validates that token against Keystone (or a cache of Keystone) and applies a local policy file that reads the roles and the scope out of it.

The proposal separates the three things the token conflates:

The central move is the third row. Today a service asks “does the role list inside this token contain admin, and does the scope inside this token match the resource?”. Under this proposal it asks Keystone “does this principal hold the role or permission this operation requires, on this target?”. The token stops being the carrier of the answer and becomes only the proof of who is asking.

Five further changes follow from this split:

• The OpenStack control plane authenticates with mTLS and nothing else. Every OpenStack service holds a SPIFFE identity issued by SPIRE. Service-to-service calls, the Keystone back-channel, and the Nova-to-Keystone attestation path all authenticate at the TLS layer. Service users, service passwords and X-Service-Token disappear.
• Trusts are replaced by service grants and on-behalf-of exchange (ADR 0036). The acting service is always recorded, no secret is ever delegated, and the delegation boundary is a fact of the authentication chain rather than of the token’s scope.
• Tenant workloads get platform identity without credentials. A VM or a Kubernetes node boots, attests to SPIRE with a Keystone-signed proof of its placement, receives a project-bound SVID, and calls the OpenStack APIs with it. No password, application credential or EC2 key is ever written into a guest.
• Humans authenticate with phishing-resistant methods. WebAuthn passkeys and OIDC federation are first-class; the design intent is for a CLI to authenticate with WebAuthn directly, with the device authorization grant covering headless machines and CI (§4.7).
• Keystone’s read path can be replicated, and an outage of its write path degrades the cloud rather than stopping it. Authorization decisions and credential validation are served from replicas of Keystone’s state — an optional data plane an operator deploys per failure domain (§4.4, §8) — while writes and authentication stay on the IAM control plane. Existing credentials keep being validated and authorized throughout an outage of it; what stops is the issuing of new ones, which the access-token lifetime turns into a stated bound (§8.5). A deployment that can live with the availability of the control-plane nodes queries them directly and deploys nothing more; the API is the same either way.

The payoff is a model in which a policy rule, an individual operation, or a set of operations can be bound to a human, a service or a workload identity on any target without inventing a new role for it, the way AWS STS session policies or Azure role assignments work. That is what finally makes “system scope”, “global reader”, “domain manager” and “service role” expressible without every service re-implementing the semantics.

2. Where OpenStack is today and what hurts

2.1 Tokens carry scope

A Keystone token is (user, scope, roles, expiry) sealed together. Consequences:

• A client that touches three projects holds three tokens, or re-authenticates on every switch. SDKs, Terraform providers and dashboards all carry code to manage this.
• Every service’s policy reads project_id and roles out of the token. The policy therefore answers “does this token’s scope match the resource” rather than “may this principal do this to that resource”. The two questions diverge exactly in the cases the community has struggled with for a decade: cross-project operations, cloud-wide read, and services acting on behalf of users.
• Scope is attacker-influenceable through rescope. Two 2026 advisories (OSSA-2026-005, OSSA-2026-015; see the security model) are instances of a decision keyed on token scope instead of on the immutable authentication chain.

2.2 Every request is a Keystone round trip

keystonemiddleware validates the token on each request, with a cache in front. Keystone is on the data path of every OpenStack API call and its availability bounds the availability of the whole cloud. Python Keystone is therefore already the single point of failure of every OpenStack deployment: every authentication and every token validation goes through it, and the validation cache in front of it is what makes a short outage survivable. The Rust implementation already addresses the validation half of this with offline JWT verification (ADR 0026 §6), but as long as roles are frozen into the token, offline validation also freezes authorization for the token lifetime.

This proposal does not remove the dependency on a running authority. It makes IAM more resistant than it is today, in three specific ways. Authentication is verified offline from a signed credential, so no round trip is needed to establish who is calling. Authorization is decided per request by a component that can be replicated independently of the nodes that accept writes — an optional data plane an operator deploys where the availability of the IAM control plane is not enough (§4.4, §8). And the cost of an outage of it is stated exactly in §8.5.

The round trip itself moves from token validation to an authorization decision, and by default no cross-request cache stands in front of it. Python Keystone’s caching layer has produced the most bugs in its design; this proposal replaces it with a replica of the grant state instead, not a smarter cache (§6.3). A caller that touches many projects therefore holds one credential and asks one question per request, batched, against something in its own failure domain.

2.3 System scope and the Secure RBAC goal

The community goal “Consistent and Secure Default RBAC” introduced three personas (reader, member, admin), later manager and service, on three scopes (project, domain, system). The design placed the scope in the token and asked every service to decide what a system-scoped token means for each of its project-owned resources. Services could not answer that uniformly: a system-scoped token has no project, so creating a server “as the system” had no owner, listing servers “as the system” needed new query semantics, and existing operator tooling broke. The goal was re-scoped; system scope is today avoided by most services and admin on a project remains the practical cloud administrator.

The root cause is not the persona model, and it is not system scope itself. It is that the scope was a property of the credential, so the credential had to be re-issued for each target, and a system-scoped credential arrived at a service with no project in it at all. This proposal keeps system scope — it is the right way to name cloud-level resources and cloud-wide authority — and removes it from the credential. An operator holding a system-level grant names a project target on the calls that need one and the system target on the calls that do not, without re-authenticating between them.

2.4 Global reader and “admin is global”

Two of the oldest open problems in OpenStack authorization are the same problem seen from both ends:

• Global reader: an auditor, a billing collector, or a monitoring system needs read access to every project. Today that requires either admin, a role on every project, or a system-scoped token that most services cannot interpret.
• Admin is global: admin on any project historically meant admin everywhere, because policies checked role:admin and the token scope was not compared to the resource. Fixing this per service took years and is still incomplete.

Both need the same primitive: a decision that takes the principal, the operation and the resource’s target, and a single authority that knows how a grant on the system or on a domain projects onto a project. That authority is Keystone; today it is not consulted at decision time.

2.5 Roles are the only unit of grant

The only thing Keystone can grant is a role on a target. To let one workload call one operation (rotate a secret, signal a Heat wait condition, read one metric endpoint), an operator must create a role, wire the role into every service’s policy file, and assign it. Nobody does that; instead the workload gets member or admin. Application-credential access rules were meant to address this: a rule matches (service, method, path) and narrows what one application credential may call. But narrowing is not granting — a credential created with no access rules inherits its role’s full permission set, so an access rule can only cut down a credential that is already as privileged as its role allows, never add a permission the role lacks. The rule also lives on the credential, not on a principal: it cannot be assigned to a human or a workload identity, held independently of the one credential it was created with, or inherited by anything else. It is a self-imposed filter over one credential’s own tokens, not a unit of grant.

Every hyperscaler solved this with permissions as the unit of grant and roles as named permission sets: AWS IAM actions and policies, GCP IAM permissions and roles, Azure actions and role definitions. OpenStack already has a cloud-wide operation vocabulary (the oslo.policy rule names such as os_compute_api:servers:create); it has never been usable as a grant.

2.6 Secrets everywhere

• Every service has a service user with a password in a config file and the admin (or service) role.
• Heat, Magnum, Mistral and others store trusts; Magnum ships a trustee password into every cluster VM, and Heat ships a per-instance EC2 keypair into the metadata of every VM that runs its CloudFormation client.
• Tenant automation gets application credentials or EC2 keys baked into images, Kubernetes secrets and CI variables, valid for months.
• Users authenticate with passwords in clouds.yaml.

Each of these is a long-lived bearer secret whose theft is indistinguishable from legitimate use.

2.7 How the rest of the industry does it

OpenStack is the outlier on every column. The proposal below moves each column to the industry pattern while keeping the v3 API and Python services working during the transition.

3. Principles

1. Authentication proves identity, nothing else. A credential says who the caller is and how strongly that was established (amr). It does not say what the caller may do.
2. Authorization is decided per request, against the target. The target is the addressed resource’s owner, or the scope the caller names for create/list operations. It is never taken from the credential.
3. Security decisions key on the authentication chain, never on scope. This is the overarching rule of the security model (of which invariant I1, “delegation facts come from the chain”, is the delegation-specific case), generalized: delegation boundaries, restrictions and the acting party are chain facts; the target is a per-call input.
4. Keystone is the authority for grants; services are the authority for resources. A service knows which project a server belongs to; Keystone knows whether the caller holds compute:servers:delete there, including through domain or system inheritance. Neither should have to know the other’s rules.
5. Credential-less wherever possible. Identity is bound to the runtime (SPIFFE SVID, attested VM, passkey), not to a secret in a file.
6. Short-lived, sender-constrained, audience-bound. Whatever bearer artifacts remain live for minutes, are bound to the presenting workload where the transport allows it, and can be narrowed to the services that may accept them.
7. Every acting party is recorded. Delegated calls carry act; audit never loses who did what on whose behalf.
8. Compatibility is a design input, not an afterthought. v3 tokens, Python services and existing SDKs keep working through every phase.
9. Grants are roles and permissions; a relation store is an implementation choice. The model works on the SQL assignment driver alone. Roles, inheritance and permission sets answer every problem in §7; an OpenFGA-backed driver (ADR 0033) is valuable for deployments that already run a relation store and for resource-level grants (§6.5), and it is a per-domain backend decision (ADR 0034), never a prerequisite for adopting anything here. The choice also decides where the data plane’s replica comes from: on the SQL and Raft drivers the data plane holds its own copy; on an externally owned store it is a client of the store’s replication (§6.3, §8.1).
10. Copies of authorization state are replicas, never caches. A replica’s freshness is maintained by the replication protocol, which observes every write, and is reported as a version with every answer. A cache’s freshness depends on the writer remembering to invalidate it, and on a lifetime when it does not. Python Keystone’s caching bugs were of the second kind, and this proposal admits only the first on the request path: a Raft learner, a database standby, or a store’s own replica. The only memo the architecture itself contains is scoped to one request and dies with it; a client-side cache is an opt-in a deployment takes on with its constraints, never part of the path specified here (§6.3).

4. Target architecture

flowchart LR
    subgraph callers["Callers"]
        human["Human<br/>passkey, OIDC, device flow"]
        ext["External automation<br/>GitHub, GitLab, Zuul, k8s operator"]
        vm["Tenant workload<br/>VM or cluster node with SVID"]
    end

    subgraph ks["IAM control plane: writes, authentication"]
        op["OAuth2 / OIDC provider<br/>issues scope-less JWT"]
        map["Mapping engine<br/>SVID / OIDC / JWT to principal"]
    end

    subgraph dt["IAM data plane (optional): replicas per failure domain"]
        authz["Authorization API /v4/authz/check<br/>grants, roles, permissions, inheritance<br/>JWKS, credential validation"]
    end

    subgraph cp["OpenStack control plane: SPIFFE mTLS"]
        svc["OpenStack service<br/>nova-api, neutron, ..."]
        pdp["Policy engine<br/>OPA, or oslo.policy bridge"]
        peer["Other service<br/>placement, glance, ..."]
    end

    spire["SPIRE"]

    human -->|"login"| op
    ext -->|"workload federation JWT"| map
    vm -->|"JWT-SVID"| map
    map --> op
    callers ==>|"Bearer JWT + OpenStack-Target"| svc
    svc -->|"authn facts, target, operation"| pdp
    pdp -->|"check(sub, op, target)<br/>per request, batched, versioned"| authz
    ks -.->|"replication: Raft learner,<br/>database standby or store replica"| authz
    svc -->|"mTLS peer identity,<br/>user JWT forwarded or exchanged"| peer
    spire -.->|"SVIDs"| cp
    spire -.->|"SVIDs"| vm

4.1 Identity tracks

4.2 The OpenStack control plane: mTLS, no dedicated authentication

Audience: deployers and service maintainers. Skip if you only need the operator-level picture; the TL;DR and §1 cover it.

Every OpenStack control-plane process holds a SPIFFE identity from a local SPIRE agent (SPIRE plan Phase 1). Keystone’s internal interface, and the internal endpoints of every other service, terminate SPIFFE mTLS and identify the peer from the SVID URI. A request from spiffe://td/service/nova-api to Neutron is authenticated by the handshake; Neutron’s policy input carries the peer as service_credentials and the forwarded user context (if any) as credentials.

This removes:

• service passwords in configuration files, and the admin role assignments services hold today;
• X-Service-Token and service_token_roles_required, which exist only because the transport carried no service identity;
• the plaintext back-channel from keystonemiddleware to Keystone (SPIRE plan Phase 4.1, the middleware SPIFFE mTLS transport change).

It does not remove the service as a subject of authorization. Every service still corresponds to a principal in Keystone — mapped from its SPIFFE ID through the mapping engine rather than carrying a password — and still holds a grant, namely the service permission set on the system target (§7.4). “No dedicated authentication in the control plane” means the transport authenticates the service; it does not mean the service is unauthorized, and it does not mean the service inherits the user’s authority. mTLS identifies the service, never the user it serves: a user-initiated call still carries the user’s JWT, and a policy that needs the user must read credentials, not service_credentials. Read the other way, this would reintroduce the service-user-with-admin pattern under a new name.

Per-host identities (.../nova-compute/host/{hostname}) let Keystone bind an attestation request to the compute host that actually runs the instance (ADR 0032), which a flat service identity cannot do.

flowchart LR
    subgraph host_a["Controller"]
        na["nova-api<br/>spiffe://td/service/nova-api"]
        ne["neutron-server<br/>spiffe://td/service/neutron"]
        ks["keystone internal<br/>spiffe://td/service/keystone"]
    end
    subgraph host_b["Compute host c-3"]
        nc["nova-compute<br/>spiffe://td/service/nova-compute/host/c-3"]
        ag["spire-agent"]
    end
    ss["spire-server"]
    na <-->|"mTLS: user JWT forwarded,<br/>peer = nova-api"| ne
    na <-->|"mTLS"| ks
    nc -->|"mTLS: POST /v4/vendordata"| ks
    ag -->|"node attestation"| ss
    ss -.->|"SVIDs"| na
    ss -.->|"SVIDs"| ne
    ss -.->|"SVIDs"| ks
    ss -.->|"SVIDs"| ag

4.3 Public API: a JWT that says who, and a request that says where

Keystone is an OAuth2/OIDC provider (ADR 0026). The access token it issues for OpenStack APIs carries:

• iss, sub, aud, exp of about fifteen minutes, jti, amr. ADR 0036 §7.1 defaults aud to the control-plane-wide openstack-apis (the domain is carried by iss, not aud), narrowed to specific service types (openstack-apis:compute, …) only when the client requests it. When no audience is requested every service may accept the token; narrowing is a way for a client to restrict a token to the services it names, never a precondition for getting one. This vision keeps that rule unchanged. It means a default token is only as confined as its sender constraint (§5.4), and that the §4.8 forwarding rule confines a token only when the client narrowed it;
• act when a service acts on behalf of the subject;
• delegation_context when the chain is delegated (service grant, application credential, on-behalf-of);
• restrictions when the holder asked for a narrower token than its full authority (a self-imposed ceiling, see §5.4);
• no project, domain or system scope, and no roles.

The caller names the target on each request. The proposal generalizes the idea behind the v3 X.509 tokenless authorization headers, which already let a client say “I am this certificate, acting on this project”, into one structured header sent on every call:

GET /v2.1/servers HTTP/1.1
Authorization: Bearer <JWT>
OpenStack-Target: project="8e3f..."

Section 5 specifies the contract. Section 5.3 shows how keystonemiddleware can turn it into the X-Project-Id / X-Roles context today’s services expect, and why that compatibility mode is a per-service opt-in rather than a free ride: a client-supplied target is not a Keystone-attested one, and a service that treats it as attested is exactly the bug class §5.4 and the security model exist to prevent.

4.4 Authorization: the policy engine asks Keystone

Each service evaluates policy per request with an input of three parts:

{
  "credentials": {
    "sub": "...",
    "amr": ["webauthn"],
    "act": null,
    "delegation_context": null,
    "restrictions": null
  },
  "service_credentials": { "spiffe_id": "spiffe://td/service/nova-api" },
  "target": {
    "scope": { "project_id": "8e3f..." },
    "server": { "id": "...", "project_id": "8e3f..." }
  },
  "operation": "compute:servers:delete"
}

The policy decides using local rules (ownership, resource state, amr requirements, delegation tripwires) and one remote fact it cannot know itself: whether sub holds the operation, or a role that implies it, on target.scope. It asks Keystone:

POST /v4/authz/check
{
  "subject": { "credential": "<the JWT exactly as presented>" },
  "target": { "project_id": "8e3f..." },
  "operations": ["compute:servers:delete"]
}

{
  "results": {
    "compute:servers:delete": {
      "allowed": true,
      "via": ["role:admin@domain:inherited"]
    }
  },
  "roles": ["admin"],
  "version": { "assignment": "raft:184223", "identity": "pg:0/3A2F18C0" },
  "lag_ms": 12
}

Keystone resolves direct, group, inherited and system-level grants, intersects with the delegation boundary and with any self-imposed restriction, and returns a decision plus the classic role list for policies that still key on roles. The boundary and the restriction are derived from the credential the request carries, not from fields the calling service filled in — they are chain facts; §6.3 sets out why a data plane that takes the caller’s word for them would reproduce the §2.1 defect class.

Every answer carries the version of the state it was decided against and how far that state was behind the write path when the answer was produced. There is no lifetime in the response and no cache in front of it: an enforcement point uses the answer for the request that asked and discards it. §6.3 sets out the freshness contract — the version, the min_version a caller may demand, and the max_staleness an operator sets — and why it is the same on every driver that lets Keystone hold a replica, and different on one that does not.

There is no cross-request decision cache, and that is the architecture rather than a posture. The live call happens on every request, batched per request, against a replica in the caller’s own failure domain (or against the Keystone nodes themselves, where no data plane is deployed). What that costs is one round trip per inbound request — the round trip today’s deployments already pay to memcached for a validation-cache hit, against a component whose answer is microseconds of in-memory work — and a hard dependency on that replica set being reachable, which §8 turns into a replica-count requirement rather than a grace window.

This is the Kubernetes SubjectAccessReview pattern, and the shape oslo.policy already supports through its http check type, which gives Python services a zero-code migration path (see the roadmap).

sequenceDiagram
    participant C as Client
    participant M as keystonemiddleware
    participant S as nova-api
    participant P as Policy engine
    participant K as Keystone authz API

    C->>M: DELETE /servers/{id}<br/>Authorization: Bearer JWT<br/>OpenStack-Target: project=P
    M->>M: verify JWT offline (JWKS cache)<br/>parse OpenStack-Target
    M->>S: request + authn facts + requested scope
    S->>S: load server, owner = P'
    S->>P: input {credentials, target{scope:P, server{project:P'}}, operation}
    P->>P: requested scope must equal resource owner (P == P') else deny
    P->>K: POST /v4/authz/check (credential, P, compute:servers:delete)<br/>(batched per request, no cache)
    K-->>P: allowed via role:admin@domain:inherited, version
    P-->>S: allow
    S-->>C: 204

Where a decision is evaluated and where the grant facts come from are two independent choices, and this proposal fixes only the second. The policy engine may run as a sidecar beside each service or as an in-process oslo.policy library; a per-service OPA sidecar calling the check API over SVID mTLS is a shape an earlier proof of concept exercised (the commented http.send stanza in policy/identity/user/list.rego). What the proposal does rule out is on the other axis: replicating the grant graph itself to every evaluation point (§6.3).

The check API is served by the IAM data plane, and the data plane is optional

Audience: implementers and operators sizing availability. Skip if you only want the outage consequences; see §8.5.

/v4/authz/check carries at least one decision per API call across the whole cloud, while grant writes and token issuance carry a load orders of magnitude smaller. Those two have no reason to share a process, a release cadence or a scaling unit, and above a certain size they must not share a failure domain either. The check API is therefore served by a component in its own right: the IAM data plane, a second binary built from the same crates as keystone (keystone-dataplane in the Rust implementation), every driver of which is pointed at a replica and whose write handlers are not built in. §8 specifies it, down to the storage work it needs first (§8.3).

It is optional. The same /v4/authz/check, JWKS and validation endpoints are served by the IAM control plane itself — the Raft voters, over the primary database — so a smaller cloud, or one whose operators accept that the availability of the IAM control plane bounds the availability of authorization, points its services there and deploys nothing further. That is the dependency such a cloud already has on Python Keystone today, with offline credential validation on top. The data plane is how an operator buys authorization availability exceeding the write path’s, per failure domain, when that is worth a replica set and a standby database. Nothing in the API differs between the two; only version and lag in the response tell a caller which it is talking to.

What the data plane is not matters as much: it never authenticates. Token issuance, refresh, login, service-grant redemption, on-behalf-of exchange and attestation are all writes, or need a signing key a replica must not hold (§8.2), and they stay on the IAM control plane. The data plane decides authorization from state it already holds and validates presented JWTs against that same state; §8.5 prices the outage case.

This is the structural lesson from Zanzibar. Google did not add a check API to its identity service; it built a separately scaled system, because deciding every request is a different engineering problem from authenticating callers. The split is also the containment boundary for everything §6.3 makes driver-dependent — batch expansion, the per-item short-circuit, the version each driver reports, the limits an externally owned store imposes — so oslo.policy and the services see one uniform contract whatever the assignment driver underneath is. Without that boundary the per-driver differences leak into every enforcement point in the cloud.

One data plane does not mean one cluster. The serving layer is federated — a replica set per region or availability zone, each independently scalable and independently failable — over a single authoritative write path, so there is still exactly one place a grant is defined and revoked. Learners do not vote, so a WAN between the IAM control plane and a regional replica set never touches commit latency, and fan-out from the leader is bounded by the replica count, tens rather than thousands — which is why replicas are per failure domain and not per host. It is not uniform across drivers: a learner or a database standby follows the SQL and Raft drivers anywhere, while a region serving a domain whose assignments live in a relation store needs a replica of that store in the region too, or pays a WAN round trip on every check (§6.3).

A partition then has a stated bound: a cut-off region keeps serving decisions from local state and stops receiving revocations, and it knows by how much — past the operator’s max_staleness it refuses to allow rather than serve an answer nobody can vouch for (§6.3, §8.1).

Three properties are requirements on the data plane rather than inherited assumptions:

• It is a second trust boundary over assignment data. The rule that security decisions key on the authentication chain and never on scope, the delegation invariants (I2/I4), and the requirement that secrets be stripped from policy input all hold inside the data plane, and are tested there with the same suites the IAM control plane runs.
• It serves from state it holds locally for as long as the write path is unreachable, including cold-booting a replacement replica while the write path is down — the statically stable posture §8.1 states as a requirement, §8.3 lists the storage work for, and §8.6 names a test for.
• Keystone’s own authorization stays in-process (§6.3). The data plane is how other services ask; it is not in the path of Keystone authorizing a call to itself, and it must not become a startup dependency of the IAM control plane.

4.5 Delegation without secrets

ADR 0036 replaces trusts with two mechanisms on the OAuth2 token endpoint:

• Service grants for deferred authority (Heat stacks, Magnum clusters, Mistral workflows): a user-visible, revocable record naming a registered service client as grantee; redeemed with the service’s SVID; token sub is the grantor and act is the service.
• On-behalf-of exchange (RFC 8693) for continuing a user-initiated operation past token expiry (Glance import, long migrations): the service exchanges the user’s token for one that carries act, bound to the root token’s jti so revoking the user’s token ends the chain.

No impersonation flag, no redelegation, no use counter, no trustee user, no password in a cluster. Both mechanisms already place every delegation fact in delegation_context, so they survive scope-less tokens unchanged (ADR 0036 §7).

A service and the guests it provisions are different identity tracks. The conductor of a Magnum cluster holds a grant here; the cluster’s nodes are §4.6 workloads with their own SVIDs. The two halves migrate independently of each other, and each has its own precondition: the trustee password can leave a guest only once the §4.6 SVID path works — the SVID is its replacement — but it does not require the grant to exist, and replacing the conductor’s trust does not require the guest to be SPIRE-attested.

sequenceDiagram
    participant U as User
    participant H as heat-api
    participant K as Keystone
    participant E as heat-engine
    participant N as nova-api

    U->>H: create stack (Bearer JWT)
    H->>K: POST /v4/service-grants (grantee = heat client, project P, roles)
    K-->>H: grant_id
    Note over E: hours later, autoscaling fires
    E->>K: POST /v4/oauth2/{d}/token grant_type=service-grant<br/>mTLS with heat-engine SVID
    K->>K: roles = grant ∩ grantor live roles on P
    K-->>E: JWT sub=user, act=heat-engine, delegation_context{grant_id, P}
    E->>N: POST /servers  Bearer JWT  OpenStack-Target: project=P
    N->>K: authz check (sub, P, compute:servers:create) with delegation bound
    K-->>N: allowed

4.6 Tenant workload identity

The SPIRE plan gives every VM a platform identity without a secret in the guest:

1. Nova asks Keystone (over mTLS, with its per-host SVID) for a short-lived attestation JWT for (project, instance); Keystone verifies placement with Nova before signing (ADR 0032).
2. The JWT is delivered through vendor data; the guest’s SPIRE agent presents it to the SPIRE server, which mints spiffe://td/project/{p}/instance/{i}.
3. The workload calls Keystone’s token endpoint with the JWT-SVID (private_key_jwt) or the OpenStack APIs directly with a JWT-SVID mapped by a rule on spiffe.project_id, and receives whatever the mapping rule grants: a role, or, with §6, a specific set of operations on its own project.

The same path serves Magnum cluster nodes, Trove database guests and any tenant automation that runs on the cloud; Octavia amphorae are a candidate once their certificate bootstrap is reconsidered (out of scope in ADR 0036). Where a service provisions guests today with an impersonating trust — Magnum writes a per-cluster trustee password and trust id into every node, and the components inside redeem the trust as the cluster owner — this path changes the actor: the guest acts as itself, a project-bound workload principal, with whatever authority the mapping rule grants. The narrowing from the owner’s roles is deliberate; a component that needs more gets a wider mapping rule or permission set, never a secret in the guest. Resources the guest creates are owned by its workload principal, not by the cluster owner. Application credentials and EC2 keys remain for automation that runs outside the cloud and cannot attest, and they shrink to a legacy niche.

sequenceDiagram
    participant NC as nova-compute (host SVID)
    participant K as Keystone
    participant VM as Guest (cloud-init + spire-agent)
    participant SS as SPIRE server
    participant API as OpenStack API

    NC->>K: POST /v4/vendordata {project, instance} (mTLS)
    K->>K: verify placement with Nova, sign with attestation key
    K-->>NC: attestation JWT (5 min)
    NC->>VM: vendor_data.json
    VM->>SS: attest with JWT
    SS->>K: GET /v4/spiffe/{domain}/jwks
    SS-->>VM: SVID spiffe://td/project/P/instance/I
    VM->>API: Bearer JWT-SVID, OpenStack-Target: project=P
    API->>K: authz check (principal mapped from spiffe.project_id, P, op)

4.7 Human authentication

• Passkeys (ADR 0005) are the default strong method. Keystone acts as the WebAuthn relying party; the OIDC provider records amr: ["webauthn"], so a policy can require phishing-resistant authentication for sensitive operations (key rotation, role assignment, grant creation).
• External IdPs federate through the mapping engine (ADR 0020), including expiring group membership (ADR 0013) and SCIM provisioning (ADR 0024).
• CLI login is meant to reach passkeys: a CLI authenticating with WebAuthn directly, with the device authorization grant covering headless machines and CI by delegating to a passkey-capable browser login, is the target design — no such CLI exists in this workspace yet. This is where phishing-resistant authentication would earn its keep, because OpenStack users authenticate at the CLI and in CI far more than at Horizon. With that path in place the clouds.yaml password becomes optional; today it is not yet documented as deprecated.
• Step-up is expressed as amr requirements in policy rather than as separate credentials.

4.8 Cross-service calls on behalf of a user

When Nova calls Neutron for a user it either forwards the user’s JWT or performs an on-behalf-of exchange (§4.5). Forwarding is simpler and works with today’s code, but a forwarded token is accepted by whatever its audience allows, so a compromised service can act as the user everywhere the token is accepted. The rule is therefore:

Forward only a token whose aud includes the callee; otherwise exchange.

A token with the default audience includes every callee, so the rule confines forwarding only for clients that requested a narrower audience (§4.3) — which is what a client holding broad authority should do. For unnarrowed tokens the exchange path is the confinement: a service that does not want to hand its callee the user’s full token exchanges it instead. Both paths carry the caller’s SVID as service_credentials, which is the control-plane identity that replaces X-Service-Token.

5. Scope-less tokens and per-request target scope

Audience: service maintainers, keystonemiddleware and SDK authors. Skip if you only need the effect on callers; the TL;DR covers it, and §5.4 is the part reviewers should read.

5.1 Request contract

A request to an OpenStack service names its target in one of three ways, in this order of precedence:

1. Resource-derived. For any operation on an existing resource (GET/PUT/PATCH/DELETE .../servers/{id}), the target is the owner of that resource as recorded by the service. This is authoritative. A client-supplied scope that disagrees with it is a policy violation and the request fails closed (404 or 403 per the service’s existing disclosure rules).

2. Header. For operations that have no existing resource (create, list, actions on collections), the client sends a structured header (RFC 8941 dictionary syntax) naming the target:

   OpenStack-Target: project="8e3f..."
   

   Exactly one target must be named, by id or by name, as project, domain or system-scope:

   Member	Value	Target
   project	project id	that project
   domain	domain id	that domain
   system-scope	all	the system: cloud-level resources, and cloud-wide reach over projects

   system-scope="all" is deliberately the same system scope the v3 API already has, spelled the way keystonemiddleware already exposes it (OpenStack-System-Scope: all). This proposal introduces no new scope type: a service that understands system scope today understands this header, and a service that does not is in exactly the position it is in today. What changes is only that the scope arrives per request instead of frozen into the credential, so an operator no longer re-authenticates to move between system scope and a project.

   Name-based addressing mirrors the tokenless X-Project-Name / X-Project-Domain-Name pair as dictionary members the middleware resolves to ids:

   OpenStack-Target: project-name="demo", domain-name="Default"
   

   The header may also carry the addressed resource, in the same <service-type>:<resource>:<id> form the operation vocabulary uses (§6.1):

   OpenStack-Target: project="8e3f...", resource="compute:server:550e..."
   

   For an existing resource the resource member never overrides rule 1: it lets the middleware and the policy engine pre-check and audit the request before the service loads the resource, and it is the hook for resource-level grants (§6.5). A resource member that disagrees with the URL, or a project that disagrees with the loaded resource’s owner, fails closed.

3. Default. If neither is present, keystonemiddleware rejects the request unless the operator has enabled fallback to the subject’s default project (an existing Keystone user attribute) for the migration period. Reject is the default; fallback is a documented, per-service opt-in, and it is time-boxed by a deprecation cycle so that it expires with the migration rather than becoming a permanent mode.

Unauthenticated endpoints are outside this contract: version discovery, JWKS, .well-known documents and health checks carry neither a subject nor a target and are never subject to rule 3’s rejection. A service’s list of unauthenticated routes is the same list it maintains today.

Using a structured field rather than a family of X-* headers keeps the grammar extensible (a later region or resource-version member is additive), gives every SDK a standard parser, and avoids the X-Project-Id family that keystonemiddleware overwrites on the WSGI environment.

The resource-derived target always wins (invariant). Rule 1 is not a preference, it is the invariant that makes the whole design safe. If a service trusted OpenStack-Target for DELETE /servers/{id}, a caller with rights on project A could name A while deleting a server owned by B. Every service that participates must therefore compare the requested target to the loaded resource’s owner and fail closed on a mismatch, with the same scope-drift tripwire discipline the security model applies to delegation (I2/I3). A service that has not implemented this comparison is not ready to accept per-request targets, which is why §5.3 makes the compatibility mode an opt-in.

5.2 What keystonemiddleware does

1. Verifies the JWT offline (signature, iss, aud, exp, nbf), or validates a legacy Fernet token through the mTLS back-channel.
2. Parses OpenStack-Target; resolves names to ids; validates that the named project or domain exists and is enabled (a read against the same replica set the check goes to). The result is the requested target, and the middleware labels it as such: it is a client assertion, not a Keystone-attested fact.
3. Populates the WSGI environment with the authentication facts and the requested target: X-User-Id, X-Requested-Target, X-Actor-* for act, the delegation context and the restrictions.
4. Leaves the decision to the service’s policy, which resolves the target of §5.1 (resource owner first) and asks /v4/authz/check for the concrete operation on the resolved target. This is the model: the policy asks Keystone whether the actor holds the required role or permission on the target, instead of reading a role list out of the credential.

The middleware never fills X-Project-Id / X-Roles from the requested target on its own initiative, because a role list computed for a client-asserted scope, placed in the variable legacy policy treats as attested, is precisely the confusion of §2.1. It fills them only in the §5.3 compatibility mode.

5.3 Compatibility

• A v3 scoped token keeps working: the middleware treats its embedded scope as the requested scope and rejects a conflicting OpenStack-Target header. A scoped token’s scope is Keystone-attested, so nothing below applies to it.

• A client that never sends the header and holds a JWT gets the default behaviour of §5.1 item 3, which an operator can set to “subject’s default project” to keep unmodified tooling working.

• Keystone continues to issue scoped tokens on request; audience narrowing (ADR 0036 §7.1) and the restrictions claim (§5.4) let a client voluntarily pre-bind a token where a deployment wants that.

• Compatibility mode is a per-service opt-in. A service can ask the middleware to keep populating X-Project-Id, X-Domain-Id, X-Roles and OpenStack-System-Scope from the requested target plus a role-list lookup, so that unmodified oslo.policy rules keep evaluating. Enabling it is a declaration by that service that it enforces the §5.1 invariant: every operation on an existing resource compares the requested target to the resource’s own owner and fails closed on a mismatch. Until that declaration the middleware refuses the mode, and the service sees only the §5.2 environment.

  The declaration is the whole point of the opt-in. The environment variables are byte-identical to today’s, but their provenance is not: under a scoped token X-Project-Id was attested by Keystone, under a per-request target it is chosen by the caller. A rule that reads project_id without comparing it to the resource — or code that stamps ownership, charges quota or shortcuts on is_admin from it — becomes caller-influenceable, which is the defect class of OSSA-2026-015. Adopting per-request targets is therefore a real, reviewable change in each service, and the roadmap budgets for it.

  The mode exists so a service can accept per-request targets before moving its policy to the check API, and is time-boxed on the same footing as the §5.1 item 3 fallback: a per-service configuration flag Keystone can report as a capability, switched off service by service (roadmap, policy-engine path).

• SDKs add one header and drop per-project re-authentication.

5.4 Threat model changes

A scope-less token represents the subject’s full authority, as an AWS access key or an Azure token does. The mitigations are the ones those platforms use, and most already exist in the ADRs. The first three are gates on shipping scope-less tokens at all, not later refinements. (Without them a scope-less bearer token is a cloud-wide master key, weaker than the scoped Fernet token it replaces.) The roadmap splits the migration at exactly this line.

• Sender constraint (gate). JWTs issued to SVID holders are presented over mTLS and bound to the SPIFFE identity by verifying the TLS peer, not by the sub claim (which is the mapped principal, not the SPIFFE ID); JWTs issued to public clients use DPoP (ADR 0026 v1.5) so a stolen token is unusable without the client’s key. ADR 0036 rules out RFC 8705 because SPIRE rotates certificates, which is correct for a thumbprint binding — the workable substitute is to bind to the SPIFFE ID rather than to the certificate, verified at the internal interface where the SVID is the TLS peer, and that binding needs specifying before scope-less tokens ship.
• Audience narrowing (gate). Available end to end, not the default (§4.3): a client may request a per-service-type audience (ADR 0036 §7.1), and services know their own type and accept it, so a token captured at one service cannot be replayed at another. A client that omits the audience gets a token every service accepts; holders of broad authority (CI, automation) are expected to narrow, and sender constraint is what protects the default token.
• Self-imposed restrictions (gate). A client may request a token whose restrictions claim limits it to named targets and operations (the AWS STS session-policy shape). Restrictions only ever narrow, are chain facts, and are intersected by /v4/authz/check. Token restrictions already exist in the Rust implementation (TokenRestriction, designed into ADR 0015’s role-mapping schema for the Kubernetes authentication method, though the k8s-auth drivers don’t wire it in yet); this generalizes them. They are also the feature application-credential access rules only ever promised for one credential type (§2.5): a CI job or a script can hold a token that can do exactly one thing. Scope-less must be the default, never the only option; without restrictions the proposal trades one problem (too many tokens) for another (every token is a master key).
• Short lifetime (fifteen minutes) with refresh-token rotation and family breach detection (ADR 0026 §9).
• Fail-closed scope agreement. The resource-derived target always wins over the header; a mismatch is an error, never a silent widening (§5.1).
• Audit. Every decision logs subject, act, requested scope, resolved target, operation, outcome and the state version it was decided against ( ADR 0023), which is what makes anomaly detection on a stolen token possible at all. Because no decision is answered from a cache, every one reaches the data plane (or the Keystone node serving in its place), whose decision log is therefore complete; the enforcement point records the same facts for correlation with the request it served (§6.3).

6. Authorization model

6.1 Roles stay; permissions are added

Keystone keeps roles, role assignments, implied roles and inheritance. It adds permissions: operation identifiers drawn from the cloud-wide policy vocabulary that every service already publishes through oslo.policy (os_compute_api:servers:create, get_image, create_port). A role becomes, in addition to its name, a named set of permissions; the SRBAC personas are shipped as such sets:

A grant is (principal, role | permission set | permission, target). Targets are system, domain:<id>, project:<id> and, later, individual resources. Inheritance is what it is today: a grant on a domain or on the system can be marked inherited and projects onto every project below it.

Which operations belong to member rather than manager in each service stays a per-project decision, and it is the same decision the SRBAC goal already requires; this proposal changes how the answer is stored, not what it is.

The operation vocabulary is a prerequisite

Per-operation grants need stable, cloud-wide operation identifiers. The oslo.policy rule names are close but inconsistent across projects (os_compute_api:servers:create versus create_port versus add_image), so the deliverable is a registry with a normalized form (<service-type>:<resource>:<verb>) and a per-project alias table. This is the largest cross-project ask in this document; authoring it is documentation and a generator rather than code.

Operating it needs more. Because /v4/authz/check expands roles and permission sets into operations at decision time (§6.3), Keystone holds a live catalog spanning every deployed service, out-of-tree ones included, that changes on every service upgrade. Four rules make that tractable:

• The catalog is deployment state with a version. Services register their operations (or an operator imports the generated registry). A vocabulary change is a change to a decision input like any other, so a replica evaluates against a stale vocabulary for no longer than its replication lag, and the registry version is part of the version it reports (§6.3).
• Unknown operations fail closed. A check for an operation the catalog does not know is denied, never allowed. A service that ships a new operation before the registry knows it is therefore broken loudly for that operation only, rather than quietly permissive.
• Persona sets are expressed as patterns, not enumerations. reader is “every *:list, *:show, *:get” evaluated against the catalog, not a frozen list of operation names, so a new read operation is covered on the day it is registered. This is what keeps a service upgrade from silently demoting existing readers, and it is why the verb half of the normalized form has to be a closed set.
• Registering an operation is a privileged, audited write. Because personas are patterns over the catalog, a registration is a grant change: an operation registered under a read verb is granted to every reader in the cloud, global readers on the system target included, without any grant being written anywhere. The closed verb set constrains the name, not the behaviour, so the registry needs what a policy rollout needs — a named owner, an audited write path, and review that the verb matches what the operation does. A service that registers a mutating or secret-disclosing operation as *:show has granted it to every auditor in the deployment. §10, question 2 asks where the registry lives; who may write to it is the other half of that question.

6.2 The authorization input contract

The community-facing deliverable is a schema, not a tool. Any policy engine (OPA is the reference; oslo.policy with an http check is the bridge) must receive:

• credentials: the authentication facts from the JWT or SVID, exactly as projected by Credentials in the Rust implementation, without roles;
• service_credentials: the mTLS peer, when a service is calling;
• target.scope: the resolved target of §5.1;
• target.<resource> and existing.<resource> per ADR 0002;
• operation: the permission identifier;
• context: extra resource facts the calling service attaches for a single decision — Neutron’s rbac_policy sharing rows, an image’s member list, a secret’s ACL, a network’s address-scope — passed straight through to the engine in its native shape (OPA input, OpenFGA contextual tuples plus a context object, Cedar context). This is how a service moves a complex, service-specific condition into the check call instead of keeping a local enforcement path for it. context feeds the decision only; it never feeds the delegation boundary, the restrictions filter or the role set, all of which stay bound to the authentication chain (I1/I2) and are not caller-supplied.

Rules that today read input.credentials.roles keep working in the compatibility mode of §5.3, where the middleware pre-fetches the role list for the requested target; that mode carries the §5.1 obligation with it. Rules that need finer control, and every rule in a service that has moved off the compatibility mode, call the check API for the specific operation on the resolved target.

6.3 The Keystone authorization API

Audience: core maintainers and implementers (the API contract). Skip if you only need what it solves; see §7.

POST /v4/authz/check (batchable) answers, for a subject and a target, which of the listed operations are allowed and through which grant. It applies:

1. direct, group-derived and inherited grants on the target;
2. system-level grants marked as applying to every project (this is how a global reader is expressed);
3. the delegation boundary from delegation_context (I2/I4 of the security model): a delegated subject can only be allowed on its delegated project and only for the delegated roles or permissions;
4. restrictions from the token;
5. implied-role expansion and role-to-permission expansion.

It is served by the IAM data plane of §4.4 where one is deployed, and by the IAM control plane itself where one is not; no caller holds a cross-request copy of the answer. The OpenFGA assignment driver ( ADR 0033) and per-domain drivers (ADR 0034) mean a deployment can back this API with a Zanzibar-style relation store for tenants that already run one, with relation sync (ADR 0035) keeping group membership consistent, at the cost of a freshness guarantee the store defines rather than Keystone, which the rest of this section works through.

The design detail of this API is in IAM data plane and check API: design detail. Its conclusions are these:

• The chain is verified, not asserted. The check request carries the subject’s credential as presented, and the data plane derives act, delegation_context and restrictions from the verified claims. A check whose chain cannot be verified is denied. Parsed fields are advisory only.
• The API is not an oracle. A caller asks about a subject presenting to it, within a target range bounded by its own reach. Enumeration is measured.
• It is the new hot path. It is batchable, versioned, admission-controlled per calling SVID, internal-only, and answered from a replica without a round trip to the write path.
• Replicas, not caches (principle 10). Freshness comes from replication, reported as a version with every answer, with min_version and max_staleness as the caller’s controls. The replica comes from a Raft learner, a database standby or an external store’s own replication, per assignment driver.
• Capacity is per caller class, and the only memo in the architecture is scoped to one request.

6.4 Binding operations to workload identities

Audience: mapping-engine implementers and operators writing rules. Skip if you do not write mapping rules; §4.6 covers the flow.

With permissions as a unit of grant, a mapping rule (ADR 0020) can bind an identity to exactly what it needs. ADR 0020’s normative rule shape is match.all_of / {type, claim, value}, an identity block (with identity_mode) and an authorizations list; the SPIRE integration plan uses the same shape. This example follows it, with two extensions the mapping engine does not have yet: a permissions authorization type for the permission-set future of §6.1, and ${claims.*} interpolation in a project authorization, which the engine copies verbatim today:

{
  "name": "cluster-autoscaler",
  "match": {
    "all_of": [
      {
        "type": "prefix",
        "claim": "spiffe.id",
        "value": "spiffe://td/project/"
      }
    ]
  },
  "identity": {
    "identity_mode": "ephemeral",
    "user_name": "${claims.spiffe.id}"
  },
  "authorizations": [
    {
      "type": "permissions",
      "project_id": "${claims.spiffe.project_id}",
      "permissions": [
        "compute:servers:create",
        "compute:servers:delete",
        "compute:servers:list"
      ]
    }
  ]
}

No role is created, no policy file is edited in Nova, and the grant is visible and revocable in Keystone. This is the AWS STS pattern, with one difference: the permission set assumed at authentication time is an upper bound on what the call may do, not the decision itself. The decision also needs the target, which this proposal binds per request (§5.1), and the grants the subject holds on that target, which stay in Keystone. There is deliberately no mint-time document that answers the question by itself — which is why the check API exists, and why attaching a small precomputed policy to the credential is not an alternative to it. Human users get the same through service grants with a role-id ceiling (ADR 0036 §4, ServiceGrant.role_ids) — extended to carry permission sets once permission sets exist — or through self-imposed restrictions.

6.5 Resource-level targets

Once services pass target.<resource> with a stable id, a grant can name a resource (server:<id>) rather than a project. This is how sharing a single image, a single network or a single secret with another principal becomes a Keystone grant instead of a per-service sharing API. It comes after permission sets and depends on services registering their resource types, but the check API shape does not change.

This is where a relation store earns its keep: resource-level grants are the shape OpenFGA (ADR 0033) models well, and a deployment that runs one can back this phase with it. Per principle 9 that stays a backend choice — the same grants are expressible on the SQL assignment driver, and no part of this proposal requires a relation store to be deployed.

Sharing rules that a service computes today from its own tables — Neutron’s rbac_policy entries being the largest example — do not have to become Keystone grants to move under the check API. A service can keep owning that data and pass it per call as context (§6.2): the engine evaluates the sharing condition, the service keeps no enforcement branch. Turning those rows into first-class resource-level grants is then an optional later step, taken if and when the service wants them managed and audited centrally.

6.6 Quota and unified limits

Quota is the one place outside policy where services read the token’s scope as an authoritative project id: it decides which project a new resource is charged to, and which project’s limit is enforced. Moving the scope into the request does not change what quota means, but it changes where the project id may come from. The rule follows §5.1: quota is charged to the resolved target, which for a create is the requested target after the service has accepted it, and never to a value read out of the credential. Two consequences deserve stating in the unified-limits work rather than being left to each service:

• A create under system-scope="all" has no project to charge and must be rejected; a cloud administrator creating a resource for a tenant names that tenant’s project as the target (§7.1), and quota is charged there, to the tenant — which is also what makes the operation auditable.
• A delegated call is charged to the delegated project, the same project the delegation boundary pins the decision to (I2), so a service grant cannot be used to charge one tenant’s resources to another’s quota.

7. What this solves

7.1 System scope and the SRBAC personas

The personas survive, system scope survives, and the scope moves out of the credential. An operator who holds admin on system sends OpenStack-Target: project="P" when creating a server for a tenant, and Nova asks Keystone whether that subject may compute:servers:create on P. Keystone says yes because the system grant is inherited. The created server has an owner (P), quota is charged to P (§6.6), and listing works with project="P".

Per-request targets remove the §2.3 situation — a system-scoped token arriving at a service with no project in it — so “create a server as the system” now has an owner: Nova needs no new semantics for system-scoped creates, because a create always names a project. Every service gets consistent persona semantics from one authority instead of implementing them in its own policy file.

7.2 Global reader

A reader grant on system, inherited, allows every *:list/*:show operation on every project. An auditor lists servers with OpenStack-Target: system-scope="all"; Nova checks compute:servers:list on the system target, receives allow, and runs its all-projects query — the same all-projects query it already runs for a system-scoped token today. No token per project, no admin, no new role, no new scope type (§5.1). The per-item obligations of I8 apply as in §6.3.

7.3 Admin is global

admin on project A now yields allowed: false for any target other than A, because the check is always against the resource’s own target. A cloud administrator is an explicit admin grant on system. There is no is_admin_project, no admin special-casing in policy files, and the distinction between project administrator and cloud administrator is a Keystone fact rather than a per-service convention.

7.4 Service users with admin

A control-plane service acting as itself is identified by its SVID, is mapped to a principal, and holds the service permission set on system; nothing else, and in particular no password and no admin. When it acts for a user it forwards the user’s JWT or performs an on-behalf-of exchange, and the policy sees both the user (credentials) and the service (service_credentials), so “only nova-api may call create_port with device_owner=compute:*” is a plain rule.

7.5 Trusts

Replaced as described in §4.5. The remaining trust-shaped use cases in Heat, Magnum, Mistral, Glance, Nova and Cinder each have a documented migration in ADR 0036 §10.

7.6 Cross-project operations

Live migration, image sharing, network RBAC, volume transfer and quota management all involve a caller acting across targets. With per-request targets, a single token and a single check per touched target express them without rescope. Services that today keep a service_token fallback to bypass scope can drop it.

7.7 Domain manager

manager on domain:D is a grant like any other; Keystone resolves it for OpenStack-Target: domain="D" and, when inherited, for every project in D. The domain-manager persona that has taken several cycles to plumb through Keystone policies is a permission set plus inheritance, and other services get it for free.

8. Revocation, availability and the IAM data plane

Audience: operators (outage cost, §8.5) and implementers (storage and replication, §8.1 to §8.4). Skip if you only need the adoption order; see the roadmap.

Offline JWT validation removes Keystone from the data path but freezes authorization for the token lifetime (ADR 0026 §11, SPIRE plan “Mode C”). The SPIRE plan names three keystonemiddleware postures: Mode A legacy (plaintext back-channel), Mode B SPIFFE mTLS with a per-request back-channel call (keeps instant revocation), and Mode C JWT offline validation (no back-channel, trades instant revocation for zero round trips). The proposal changes the trade-off:

• Authentication is validated offline: signature and expiry.
• Authorization is checked on every request against a replica of Keystone’s state. Disabling a user, removing a role, revoking a grant or narrowing a delegation takes effect when the replica applies the write, which is replication lag — milliseconds inside a failure domain, seconds across a WAN — and never a lifetime an operator configured. The replica reports the lag, and past max_staleness it refuses rather than allows (§6.3).
• An optional signed revocation feed (/v4/oauth2/{domain}/revocations, SPIRE plan “Mode C”) bounds credential revocation for offline validators without a per-request round trip. Where credential validation is served from the data plane, the revocation events are part of the replicated state and need no separate feed.

Keystone therefore remains a runtime dependency, as §2.2 states: this proposal makes IAM more resistant in a stated way rather than pretending to remove the dependency. Which component the request path depends on is the operator’s choice, between the IAM control plane serving everything (§4.4) and a data plane per failure domain.

In either case a check that cannot reach any serving replica fails closed: denied, with no grace window, because there is no cached allow to replay. Availability is bought with replica count and a client that tries them all, never with a stale window — a simpler posture than keystonemiddleware’s token cache and a stricter one than the Kubernetes webhook authorizer’s, and what makes the audit record complete (§6.3). A deployment wanting stale-if-error for read-only integrations enables the client-side cache opt-in of §6.3 for those services and accepts its constraints; it is not the default.

That posture is an operational requirement, so it belongs in deployment guides: fail-closed behaviour stated explicitly; at least three replicas per failure domain on separate hosts, each holding its own copy of the state; a replica set per region, because a cross-region round trip on every check puts a WAN in the middle of every API call; and capacity stated in decisions per second and per-caller-class terms (§6.3), with lag and refusal rate as alerting signals rather than dashboards.

8.1 Splitting the IAM control plane from the data plane

The split of §4.4 — an IAM control plane that mutates state and authenticates, replicas that evaluate — is the industry’s answer to this section’s problem. On its own it buys nothing on availability — one replica is the same single point of failure with a different process name — but it buys independent scaling, an independent release cadence and a containment boundary. Availability comes from three further properties, each of which has to be a requirement rather than a deployment habit.

Static stability. Every replica holds a durable local copy of the inputs it serves and keeps serving from that copy while the write path is unreachable — for the length of the outage, bounded only by max_staleness. The corollary is the one implementations lose: no startup dependency on the control plane. A replacement replica must cold-boot and begin serving with the write path down, because a replica that can only warm its state by calling the component it exists to survive is coupled to it, whatever the deployment diagram says.

The requirement binds whoever owns the inputs, which is not always the replica. Where Keystone owns them — the SQL and Raft drivers — the local copy is the replica’s own, and static stability is Keystone’s to hold and to test. Where a domain’s assignments live in an externally owned store (ADR 0033), the replica is a client rather than a holder: it has no copy to cold-boot from, and static stability becomes a property of that store’s own replication (an OpenFGA instance per failure domain over a standby of its database gives the same posture). An operator choosing that driver is choosing this alongside the freshness ceiling of principle 9, and a data plane serving both kinds of domain is statically stable for part of its traffic and not for the rest — a deployment-guide fact, not a footnote.

One failure domain per replica set. The federated serving layer of §4.4 is the definition of a supported high-availability topology, not one option among several: a replica set per region, no cross-domain call on the read path, and a partitioned region degrades alone, to the bound it declared.

One lag, operator-visible. With no lifetime at the enforcement point, the only staleness term is replication lag, and every replica measures and exports it. What an operator can promise is that number, and max_staleness is what they promise it will never exceed while an allow is served.

8.2 to 8.4 What the data plane is made of

The data plane is a second binary built from the same crates, keystone-dataplane. It is read-only by construction, serves the check API, JWKS and credential validation, and never authenticates. It needs five things from the storage and SQL layers before it can serve (local reads on learners with a version, lag metrics, seeding without the leader, a learner SPIFFE role, and a version on the SQL path). Secrets are kept away from replicas by tier-keyed encryption and, for replicas outside the control-plane trust zone, a second Raft group. The detail is in IAM data plane and check API: design detail.

8.5 What an outage of the IAM control plane costs

This is the actual answer to “Keystone must not be a single point of failure”. With a data plane deployed, a total outage of the IAM control plane costs exactly this:

• no new or changed grants, role definitions or implied-role edits;
• no operation-vocabulary changes (§6.1);
• no authentication of any kind: no login, no token issuance, no refresh, no JWT-SVID exchange, no service-grant redemption, no on-behalf-of exchange (§4.5, ADR 0036), no attestation for new instances (§4.6);
• no project, domain or user lifecycle operations.

Existing credentials keep validating and keep being authorized throughout, from the replicas, until they expire — and that last clause is the cost the token lifetime sets. With fifteen-minute access tokens (§4.3), a client whose token expires during the outage cannot obtain another until the IAM control plane returns, and a workload whose refresh rotation fails is in the same position. The data plane does not change this, by design: authentication includes writes, and a replica that issued tokens would be a signing oracle in every failure domain. What an operator can tune is the access-token and refresh-token lifetimes, which are availability parameters as well — a longer access-token lifetime buys tolerance of a longer outage at the price §5.4 states. The list above belongs in the deployment guide beside the replica-count numbers: it is what an operator is buying. Read against §2.2, where a Keystone outage stops every request the moment the validation cache expires, it is still the largest availability change in this proposal.

One further lever is already in the repository and unremarked: per-domain drivers (ADR 0034) make the write path shardable by domain, so a cloud-wide control-plane failure domain is a deployment default rather than something this architecture requires.

8.6 How other platforms answer the same problem

Every large platform that decides authorization centrally has converged on the same two answers:

Nobody keeps grant propagation synchronous, and everybody buys availability by replicating the evaluator rather than the enforcement point. None answers the availability problem by handing the enforcement point the grants: where entitlements do ride in a token, as Entra’s directory- and app-role claims do, they cover a narrow, slow-moving class of authority and carry the frozen-scope problem §2.1 exists to remove; nothing on the resource path uses that shape. Kubernetes is the one place the evaluator and the enforcement point coincide, and its in-process RBAC is therefore the grant-graph replication §6.3 refuses; the reasons it works there do not transfer, because the graph is small, a cluster is a scope boundary, and there is one enforcement point rather than one per service.

Four shapes could reduce the coupling further — two taken up, two refused:

• Replica sets per failure domain, statically stable, with lag published. The three requirements above: no new protocol, a hardening of §4.4 plus the storage work of §8.3.
• A materialized closure index inside the replica. Group and hierarchy expansion precomputed and incrementally maintained from the replication stream (§6.3). A scaling lever rather than an availability one, invisible to services, available where Keystone owns the inputs and unnecessary on a store that maintains its own. Worth reaching for when fan-out shows up in a profile.
• A node-local decision agent. Refused: a per-host daemon holding decisions is a cache in the sense of §6.3, and a replica per host puts the whole grant graph on every compute node and makes the leader replicate to thousands of learners. Replicas are per failure domain, not per host (§10, question 11).
• Entitlements carried in the token. Refused, though it has the best availability properties on this list — no runtime dependency on anything — for the reason §2.1 gives: this proposal knowingly trades the strongest availability story available for per-request authorization.

Local evaluation in every enforcement point is not on that list: §6.3 refuses it, and on an externally owned store it is unavailable at any price (§10, question 8).

Static stability decays silently when it is not exercised, so it ships with a test rather than a paragraph. The deployment guide names three game-day exercises: stop the IAM control plane and verify that the cloud still validates and still authorizes existing credentials from the replicas, and that lag climbs on every dashboard; partition one region’s replica set and verify that only that region degrades, and that it refuses at max_staleness; and cold-boot a replacement replica with the IAM control plane still down.

8.7 What a central authority costs

None of the above removes the trade this architecture makes. A single authority for mutually independent trust domains means its outage is everyone’s outage at once, and a bad grant write or policy rollout has cloud-wide and cross-tenant reach. The mechanisms here shrink the blast radius (per-class capacity, refusal past max_staleness, secrets kept away from replicas) and lengthen how long an outage is absorbed before it is felt (region-local replicas, static stability, bulkheaded pools). They do not decouple anything.

This is the bet AWS made with IAM, and it is defensible. IAM’s mitigation set is the one above, regional isolation included, plus accepting and documenting eventual consistency in grant propagation. The industry answer converges on replication lag as a normal, documented property rather than a defect. For OpenStack that is an expectation change as much as an architectural one: revocation today is nominally immediate, and this proposal trades nominal immediacy for a lag an operator can measure, read and bound.

Two consequences follow for how the work is delivered. The data plane and the policy bundle both need staged rollout and canarying, because correctness blast radius is now as cloud-wide as availability blast radius — and the policy bundle is already a named trust boundary with a half-closed supply-chain gap (doc/src/contributor/security-review.md §V4: the publish side signs and verifies its own signature in CI, while the consuming side still pulls a mutable :latest tag with no verification wired into the load path). And the bet belongs in front of operators at adoption time.

9. Adoption and coexistence

How this is adopted, in which order, and what each project has to change is in the roadmap. This section states only what must hold however the order is chosen.

Every step is additive. Python Keystone, Fernet tokens and unmodified services keep working throughout, and four rules are what “additive” means:

• v3 scoped tokens and OpenStack-Target coexist. A scoped token’s own scope is the requested scope, and a conflicting header is rejected (§5.3).
• Python Keystone keeps validating the tokens Rust Keystone issues (ADR 0026) until it is retired.
• No Python-owned table changes; new state lives in Raft-backed drivers.
• Every capability has an off switch that restores the previous behaviour for one service without redeploying Keystone.

Two properties are not negotiable, because without them the proposal is weaker than the status quo. Scope-less tokens do not ship without restrictions, audience narrowing and sender constraint (§5.4). Per-request targets are not accepted by a service until it enforces the resource-owner-wins invariant (§5.1). Everything else may be adopted per service, on the service’s own schedule, or not at all. A deployment may stop at role resolution (the roadmap’s Mode 1) indefinitely: it loses resource-level decisions and central policy change, and keeps everything it enforced before.

Held to those rules, every step delivers a system strictly stronger than the status quo. It removes long-lived secrets from the control plane and from guests, makes every delegation visible and revocable, gives every service the same persona semantics from one authority, and makes the scope and role decisions behind the last decade of authorization advisories impossible to key on the wrong input.

10. What the community still decides

1. Header grammar details. OpenStack-Target as an RFC 8941 dictionary is proposed; open are the exact member names, whether resource is mandatory for single-resource operations once services support it, and how multi-target operations (server migration between projects, image sharing) name a second target.
2. Where does the operation registry live, and who may write to it? In oslo.policy as generated documentation, in Keystone as a resource services register at startup, or in a separate governance-owned document? The second half is the sharper one: a registration is a grant change (§6.1), so an open self-registration endpoint lets any deployed service widen what every reader may do. Whether the write is operator-curated, governance-reviewed, or service-supplied with an audit trail is a decision about who holds a privilege, not where a file lives.
3. Cross-project listing for non-admins. system-scope="all" reuses the existing system scope and therefore keys on a system-level grant, exactly as today. It leaves untouched the case of a user with roles on many projects who wants one listing across them; resolving that to “the set of projects the subject can read, with the service filtering” is friendlier to those users and considerably more expensive. Is it worth a separate target, or is it a client-side concern?
4. DPoP versus mTLS-bound tokens for public clients. DPoP is the OAuth standard; OpenStack clients have no established key-management story for it yet.
5. Resource-level grants. Whether and when to let Keystone hold grants on individual resources, replacing per-service sharing APIs.
6. Retirement timeline for application credentials and EC2 keys once off-cloud automation can use workload federation and on-cloud automation has SVIDs.
7. Should driver-dependent freshness be an operator-facing capability matrix? Whether each assignment driver declares what it reports as version, whether it honours min_version, and what revocation latency it can be held to — a contract operators can hold the project to — or whether that stays an implementation note in ADR 0034, is a governance question as much as a technical one. Operators pick drivers on it, and an LDAP-backed or externally stored input is the row that reports no version at all.
8. Is anything owed to deployments that want local evaluation? This proposal leaves a live check against a replica as the only shape, rejecting both pushed entitlement snapshots and grant-graph replication (§6.3) — and on a domain backed by an external store, no other shape is available in any case. For a SQL-only deployment willing to accept that every enforcement point holds every tenant’s grants, local evaluation is technically reachable, so the question is whether that is documented as a supported option or named and refused. This proposal names and refuses it; a single-tenant deployment may reasonably disagree, and should say so here rather than build it quietly. What is not open is whether the coupling exists: §8.7 states it as an accepted cost.
9. Should the client-side cache opt-in exist at all, and who administers it? §6.3 admits it for deployments that accept a stale window and §8 keeps it out of the architecture. Whether it ships in oslo.policy, whether an operator sets it per service or a domain administrator may set it for integrations in their own domain, and what maximum lifetime a cloud can impose over that choice are governance questions with an API surface, and they should be answered before anything ships that can serve a stale allow.
10. Does the check API receive the credential, or a signed chain assertion? §6.3 requires the data plane to verify the chain facts itself and recommends the direct route, forwarding the credential as presented: simpler, strictly stronger, and disclosing nothing the callee does not already hold. It also means every enforcement point hands the user’s credential to a second component, which some deployments will read as widening its reach even though the data plane is internal-only. A Keystone-minted chain assertion avoids that and gives up liveness. Which is the default, and whether the trusted-asserter posture is offered at all, should be settled before the check API request schema freezes.
11. Is the node-local decision agent part of the target architecture? §8.6 answers no; the question is recorded so a deployment measuring a case for one argues here.
12. When does a domain move its assignments to the Raft driver? §6.3 compares the two Keystone-owned drivers. Whether a greenfield deployment starts on Raft, whether a domain migrates at Python Keystone’s retirement or earlier under ADR 0034, and whether group membership and hierarchy are mirrored into the driver or joined across providers, are decisions for that driver’s ADR.
13. Which mechanism keeps secrets away from replicas, and which secrets? §8 names tier-keyed encryption and a second Raft group and orders them. The tier of each keyspace — which of sessions, API keys, signing keys and credentials a learner may hold as ciphertext it cannot open, and which it may not hold at all — is the storage ADR’s to decide before the first replica runs outside the control-plane trust zone.

11. References

Most ADRs below are Accepted and describe the system as it stands. ADRs 0024, 0030, 0031, 0032, 0034, 0035 and 0036 are still Proposed; this vision describes them in the same present tense as the rest for readability. ADR 0024’s SCIM endpoints and ADR 0032’s vendordata endpoint already ship; the rest are not implemented yet. ADR 0036 carries the most weight here — §4.5 and §7.5 build directly on it — so its proposed status is the one to track.

• ADR 0002 — Open Policy Agent
• ADR 0005 — Passkey authentication
• ADR 0008 — Workload federation
• ADR 0013 — Expiring group membership
• ADR 0014 — Application credentials
• ADR 0017 — Security context
• ADR 0020 — Mapping engine
• ADR 0021 — API-key ingress
• ADR 0022 — Rate limiting
• ADR 0023 — Auditing
• ADR 0024 — SCIM v2 provisioning
• ADR 0025 — Dynamic authentication plugins
• ADR 0026 — OAuth2 / OIDC provider
• ADR 0030 — Per-request cache
• ADR 0031 — Prometheus metrics
• ADR 0032 — Vendor data JWT attestation
• ADR 0033 — OpenFGA assignment driver
• ADR 0034 — Per-domain assignment drivers
• ADR 0035 — Relation sync provider
• ADR 0036 — Service delegation
• ADR 0037 — Access-rule enforcement
• Security model
• Roadmap
• IAM data plane and check API: design detail
• SPIRE integration plan
• Control plane and data plane split, review and proposal
• OpenStack community goal: Consistent and Secure Default RBAC
• Keystone X.509 tokenless authorization (Python Keystone feature, not present in this repo — see the OpenStack Keystone tokenless authentication docs)
• RFC 8693 (token exchange), RFC 9449 (DPoP), RFC 8705 (mTLS-bound tokens), RFC 7523 (JWT client authentication), RFC 8941 (structured field values), SPIFFE and SPIRE specifications

Roadmap: adopting the authentication and authorization vision

Status: Draft for community discussion

Date: 2026-10-14

The vision says what the target model is and why. This document says how OpenStack gets there and in which order. It is a plan to argue about, not a commitment by any project. Section numbers written as §N refer to the vision and link to it, unless the link stays within this document. ADRs are referenced by number.

The order is driven by dependencies, not by a single line of phases. Some work is a hard prerequisite of other work, and a lot of it is independent. The graph in §2 shows which is which, so that teams can start in parallel and so that a slow item does not hold back work that never needed it.

Decisions that gate milestones. Three decisions sit outside the work itself, and a milestone that depends on one cannot start until it is made:

• Amend ADR 0016-v2 §3 so learners may serve local reads, before E1 (detail).
• Settle whether the check API receives the credential or a signed chain assertion, before the request schema freezes in C1 (open question 10).
• Decide who owns the Go SPIRE node attestor, before A2.

1. Rules that hold throughout

Every step below must respect the additive rules of the vision (§9), and this plan’s rule that each release cycle carries one large, user-visible change (the token-format change, stream B, is the largest and gets a cycle of its own).

• Python Keystone, Fernet tokens and unmodified services keep working.
• v3 scoped tokens and OpenStack-Target coexist. A conflicting header is rejected.
• No Python-owned table changes. New state lives in Raft-backed drivers.
• Every capability has an off switch that restores the previous behaviour for one service without redeploying Keystone.

2. Workstreams and dependencies

Five workstreams. Each milestone has an identifier used in the graph and in the tables that follow.

flowchart TB
    subgraph A["A: identity foundation"]
        a0["A0 SPIRE for the control plane"]
        a1["A1 secretless control plane"]
        a2["A2 tenant workload identity"]
        a0 --> a1
        a0 --> a2
    end
    subgraph B["B: token format"]
        b1["B1 JWT validation in the shared libraries"]
        b2["B2 JWT issued by default,<br/>Fernet still accepted"]
        b3["B3 Fernet validate-only, then off"]
        b1 --> b2 --> b3
    end
    subgraph C["C: authorization"]
        c1["C1 per-request target<br/>and check API"]
        c2["C2 scope-less JWT"]
        c3["C3 operation vocabulary<br/>and permissions"]
        c4["C4 policy-engine path"]
        c1 --> c2 --> c3 --> c4
    end
    subgraph D["D: delegation"]
        d1["D1 service grants<br/>and on-behalf-of"]
        d2["D2 service migrations<br/>(multi-cycle)"]
        d3["D3 trusts off,<br/>then removed"]
        d1 --> d2 --> d3
    end
    subgraph E["E: data plane (optional)"]
        e1["E1 real learners"]
        e2["E2 keystone-dataplane"]
        e3["E3 grant index<br/>and secrets isolation"]
        e1 --> e2 --> e3
    end
    a1 --> d1
    b1 --> d1
    b1 --> c2
    a1 -->|"sender constraint"| c2
    c1 -.->|"delegation-aware resolver"| d1
    a2 --> d3
    c1 --> e2
    c3 -.-> e3

Solid arrows are hard dependencies. Dotted arrows are requirements one side places on the other without blocking its start.

2.1 What the arrows mean

Two corrections to the intuitive ordering are worth stating.

• Delegation does not wait for scope-less tokens. D1 ships against scoped tokens. The reverse constraint is the one that matters: when the check API resolves roles at request time, it must compute live roles of the subject on the target, intersected with the grant’s role ids, and require the target to equal the delegated project (ADR 0036 §7). A resolver that ignores the delegation would let the target a delegated caller names widen the delegation.
• Workload identity does not wait for scope-less tokens. The SPIRE plan maps an attested VM to a project-bound principal through the mapping engine, which needs no change to token scope.

2.2 Global view by release cycle

The same milestones laid over the OpenStack six-month cycle. Cycles are named relative to the first planned one: x.1 is the first development cycle (released in April), x.2 the one after it (October), then x+1.1, x+1.2 and so on. x-1.2 is already released. The dates are indicative: they show order and size, not a commitment by any project. A stream can slip without moving the others, except along the arrows in §2. In the chart, a bar that ends at an April tick finishes in a .1 release and one that ends at an October tick in a .2 release.

gantt
    title Adoption by release cycle
    dateFormat YYYY-MM
    axisFormat %b
    tickInterval 6month
    section A identity
    A0 SPIRE in devstack and deployment tools :a0, 2026-10, 2027-04
    A1 middleware SPIFFE transport          :a1a, 2026-10, 2027-04
    A1 deployment tooling adoption          :a1b, 2027-04, 2027-10
    A2 pilot in devstack                    :a2a, 2027-04, 2028-04
    A2 general availability                 :a2b, 2028-04, 2028-10
    A2 Magnum guests credential-less        :a2c, 2028-10, 2029-04
    A2 Heat CFN guests credential-less      :a2d, 2028-10, 2029-04
    Retire app creds to off-cloud only      :aret, 2029-04, 2029-10
    section B token format
    B1 JWT validation in libraries          :b1, 2026-10, 2027-04
    B1 adopted by services                  :b1b, 2027-04, 2027-10
    B2 JWT issued by default                :crit, b2, 2027-10, 2028-04
    B3 Fernet validate-only                 :b3a, 2028-10, 2029-04
    B3 Fernet off                           :b3b, 2029-04, 2029-10
    section C authorization
    C1 Keystone side and libraries          :c1a, 2026-10, 2027-04
    C1 service adoption (Mode 1)            :c1b, 2027-04, 2028-10
    C2 gate work                            :c2g, 2027-04, 2028-04
    C2 scope-less JWT                       :crit, c2, 2028-04, 2028-10
    C3 permissions and vocabulary           :c3, 2028-04, 2029-04
    C4 policy-engine path                   :c4, 2029-04, 2029-10
    section D delegation
    D1 Keystone side and keystoneauth       :d1a, 2026-10, 2027-04
    D1 pilots (Heat, Glance)                :d1b, 2027-04, 2027-10
    D2 service migrations                   :d2, 2027-10, 2029-04
    D3 trusts off by default                :d3a, 2029-04, 2029-10
    D3 trusts removed                       :d3b, 2029-10, 2030-04
    section E data plane
    E1 real learners                        :e1, 2026-10, 2027-10
    E2 keystone-dataplane                   :e2, 2027-10, 2028-04
    E3 grant index and secrets isolation    :e3, 2028-04, 2029-04

What each release delivers

Audience: operators and release managers. Skip if you only need the order of work; the dependency graph covers it.

• x-1.2 (Oct). No change for operators or users; the design is agreed.
• x.1 (Apr). Operators can grant a service standing, bounded rights to act for a user, as a trust replacement for new setups. Libraries can validate JWT tokens and ask Keystone whether a request is allowed. SPIRE is delivered in devstack and the deployment tools (A0), and control-plane calls can use SPIFFE instead of passwords (A1 transport).
• x.2 (Oct). Heat and Glance run long operations without trusts. Services start enforcing authorization through the check API. Services stop holding Keystone passwords for token validation. VM workloads get credentials in a devstack pilot.
• x+1.1 (Apr). Users get JWT tokens by default. Services validate them locally, with less load on Keystone. Service migrations away from trusts proceed.
• x+1.2 (Oct). Scope-less tokens: one token works across projects and is narrowed per request. Tenant VM workloads get credentials without stored secrets (general availability). Fernet is deprecated.
• x+2.1 (Apr). All services have left trusts. Roles can be expressed as permissions on operations. Magnum and Heat CFN guests need no stored credentials (the per-VM EC2 keypairs leave the guest metadata). Fernet tokens validate only.
• x+2.2 (Oct). Trusts are off by default. Fernet is off. Authorization can run in the policy engine. Application credentials are meant for off-cloud clients only.
• x+3.1 (Apr). Trusts are removed from Keystone: no API, no table, no code path.

What shapes this schedule

• Libraries lead services by one cycle. keystonemiddleware, keystoneauth1 and oslo.policy are cycle-released deliverables, and a service can only consume what a released library offers. Every service-side adoption therefore sits one cycle after the library change it needs: B1 in x.1 and services in x.2, and C1 in x.1 with first services in x.2.
• Stream B gets a cycle of its own. B2 is the biggest visible change for end users. x+1.1 carries no other user-visible change. Service-side work that users do not see (D2 and C1 adoption) continues in the same cycle.
• Breaking defaults land on skip-level boundaries. Releases that allow skip-level upgrades (x.1, x+1.1, x+2.1) are where operators jump. The default flips (B2, the Fernet steps, trusts off) are placed so that each falls in a cycle with at least one full release of warning before it. The exact deprecation periods have to be checked against the current TC policy before this plan is adopted.
• D2 is the long pole. It touches Heat, Mistral, Senlin, Aodh, Sahara, Trove, Magnum and Glance, each on its own team’s schedule. Three cycles is a guess, not an estimate, and D3 cannot start before it ends. Trusts stay available throughout, so a service that is late blocks only D3, not the services that finished.
• C2 waits for its gate, not for the calendar. Restrictions, audience narrowing and sender constraint are all built in the two cycles before it. If any of them slips, C2 slips with it, and nothing else in the plan has to.
• A2 depends on a decision, not just work. The SPIRE node attestor is a Go plugin that needs a home in the community. If that is not settled by x.2, A2 and with it D3 move.
• Python Keystone. Dates assume Rust Keystone as the reference implementation. A deployment that stays on Python Keystone can adopt the library and service changes but cannot redeem grants or serve the check API until Python Keystone gains them or is retired (ADR 0036).

3. Stream A: identity foundation

A0. SPIRE for the control plane

SPIRE is the cornerstone. It is deployed for the OpenStack control plane only, not for end users, and is configured to issue an identity to every control-plane process and every compute host (SPIRE plan Phases 1 and 2).

Status: delivered in x.1, in devstack and in the deployment tools together. This is the earliest an operator can run it.

A1. A control plane without Keystone credentials

• keystonemiddleware gets a SPIFFE mTLS transport for validation (SPIRE plan Phase 4.1).
• Service-to-service calls authenticate at the TLS layer. Service users, service passwords and X-Service-Token stop being needed (§4.2).
• Legacy services that still ask Keystone for a token keep working.
• Secrets that services hardcode into provisioned VMs, such as Magnum’s and Heat’s, stay until A2 (and D3 for the trust-based ones).

The goal is “no Keystone credentials in the control plane”. Database and message queue credentials are out of scope.

A decision to make here: service-to-service calls can forward the user’s token or exchange it (D1). A1 does not need to choose, because forwarding works with mTLS alone.

A2. Tenant workload identity

Depends on A0. It does not depend on scope-less tokens.

1. Nova asks Keystone for vendor metadata (ADR 0032). The metadata contains a short-lived attestation token that Nova injects into the VM.
2. The SPIRE agent in the VM presents it to the SPIRE server, which issues a project-bound SVID. This happens on new VMs only.
3. The workload presents the JWT-SVID to Keystone, which maps it to a principal bound to spiffe.project_id (ADR 0020).

Open items:

• How the mapping gets its authority. Authority must come from what Nova attests on the server side. Metadata that the guest or user supplies must never decide roles. Narrow authority by default (the compute-vm role of the SPIRE plan) and wider authority by explicit mapping rules or permission sets (§6.4).
• The SPIRE node attestor is a Go plugin, and it lives outside OpenStack governance and release practice. Mirroring the repository solves hosting, not review, CI or releases. Ownership has to be decided before this milestone.
• Magnum is a separate migration with its own plan and the Magnum team ( ADR 0036 §10.3).
• Heat’s in-guest CFN credentials — a per-server EC2 keypair delivered in the server’s metadata, plus the stack-domain user’s password for the heat and zaqar transports — retire on this track too, once the in-guest CFN client can present the JWT-SVID (ADR 0036 §10.2). They do not gate D3.

4. Stream B: token format

This is the longest stream and the most visible to end users. It runs in a release cycle of its own.

• B1. JWT validation in the shared libraries. keystonemiddleware and keystoneauth1 can validate Keystone JWTs: signature against the JWKS, the audience, the issuer allowlist. This is the only part of the stream that other work depends on.
• B2. JWT issued by default. Keystone becomes a full OAuth2 provider. Fernet and JWT are accepted side by side, and at some point the default issuance flips to JWT.
• B3. Fernet validate-only, then off. Last, after the deployments that need Fernet have moved.

Offline validation (A1 and B1 together) accepts that revocation takes effect at token expiry unless the validator checks live (§8). The 15-minute access token is the stated bound.

5. Stream C: authorization

C1. Per-request target and the check API

Five deliverables. None changes a credential, and tokens stay scoped.

• The OpenStack-Target header, and middleware that exposes the requested target as a client assertion (§5.2).
• /v4/authz/check, served by the IAM control plane from the first release, with version on every response (§6.3).
• The oslo.policy keystone check type: request-scoped memo, replica list, fail-closed breaker, no cross-request cache.
• Per-service adoption of the resource-owner-wins invariant (§5.1), with opt-in to the compatibility mode (§5.3) and SDK header support.
• A delegation-aware resolver (see §2.1).

The operation vocabulary arrives with C3, and unknown operations fail closed from the start (§6.1). What the check API answers in C1 is role resolution on a target, plus any operations a deployment has registered.

The request schema of the check API freezes here, so the open question on whether the API receives the credential or a signed chain assertion (§10, question 10) has to be settled first.

C2. Scope-less JWT (gated)

C2 may not start until all of these are shipped:

• the restrictions claim (gate);
• service-type audience narrowing available end to end: clients can request it (ADR 0036 §7.1) and services accept their own type (§4.3; gate). It is not the default, and an unnarrowed token stays valid at every service;
• sender-constrained presentation: DPoP for public clients, SPIFFE-ID binding for SVID holders (§5.4; gate).

Then: scope-less access tokens, the JWT offline filter, and scoped tokens demoted to a compatibility option. The gate is the load-bearing part. Scope-less tokens without it are a cloud-wide master key, weaker than the token they replace.

C3. Operation vocabulary and permissions

The operation registry, with versioning and fail-closed unknown operations (§6.1). Roles become permission sets. Mapping bindings carry permissions. The batch check API serves collection reads (§6.3).

This is a separate program from C1 and C2, and the one with the most open governance questions: where the registry lives, who may write to it, and whether services push their policies or a central plane distributes them (§10, question 2). It should not hold C1 and C2 back.

C4. Policy-engine path

Services that want central authorization move from oslo.policy http checks against /v4/authz/check to a full policy-engine input. The compatibility mode of §5.3 is switched off for those services one at a time. Service passwords and service admin grants are removed where no longer used. X-Service-Token is retired once nothing depends on it.

5.1 Two ways to consume the check API

How much of the decision a service hands over is a spectrum. The first mode is a valid place to stop.

• Mode 1, role resolution. The actor and the OpenStack-Target go to Keystone on every request, which returns the effective roles and permissions on that scope. The service evaluates its existing oslo.policy rules locally, as it does today with a scoped token. The swap happens inside keystonemiddleware and oslo.policy, not in the service. The service makes a one-time, reviewable compatibility-mode declaration that its policy checks compare the requested target to each resource’s own owner (§5.3). A service whose policy.yaml tests only scope-level facts typically needs no code change to satisfy it. This is the drop-in state when tokens go scope-less in C2.
• Mode 2, full delegation. The service sends the actor, the operation and the resource and receives allow or deny. This is the end state of C4.

A deployment may stay on Mode 1 indefinitely. It gives up resource-level decisions (§6.5), permission-set personas beyond what local rules encode, and central policy change without redeploying a service. Everything each service enforced before keeps working.

6. Stream D: delegation

Design: ADR 0036.

• D1. Service grants and on-behalf-of exchange. The registry of delegatable services, the grant resource and API, the redemption grant, the exchange, lifecycle hooks and audit events. Pilots: Heat for grants, Glance for exchange. JWT-SVID client authentication on the public interface comes with its SPIRE preconditions (ADR 0036 §11, Phase 3).
• D2. Service migrations. Every service that uses trusts swaps one creation call and one authentication plugin: Heat, Mistral, Senlin, Aodh, Sahara, Trove, Magnum’s conductor, Glance. This is not a large change per service. It is a multi-cycle effort across all of them, and it is the long pole of this stream.
• D3. Trusts off, then removed. A configuration switch turns trusts off by default, with EC2 credentials under a grant handled first. Removal follows once every deployment that shares a database with Python Keystone has moved trust creation off it.

Trusts can be turned off only when D2 is complete for every service in use and A2 has removed the trustee password from Magnum guests (the dependency in §2.1). Neither C2 nor C3 is required.

7. Stream E: the IAM data plane (optional)

Design: IAM data plane and check API and the control plane and data plane split proposal. A small cloud that tolerates the availability of the Keystone nodes deploys none of this, and the API is the same either way.

• E1. Real learners. Local reads with a version, lag metrics, a learner SPIFFE role, and restore-based seeding.
• E2. keystone-dataplane. The read-only binary over a database standby and a Raft learner, serving the check API from C1 and credential validation.
• E3. Grant index and secrets isolation. The in-memory grant index, batch checks at lookup cost, and tier-keyed encryption or a second Raft group so that replicas outside the control-plane trust zone become supportable.

The amendment to ADR 0016-v2 §3 for learner reads must be made before E1.

8. Changes per project

Audience: maintainers of individual OpenStack projects. Skip if your project is not listed; the workstreams cover the rest.

9. Retirement

These happen last, and each is independent of the others.

• Fernet validate-only, then off (B3).
• /v3/OS-TRUST removed (D3).
• Application credentials limited to off-cloud automation, once workload federation covers off-cloud automation and SVIDs cover on-cloud automation (§10, question 6).

10. Where the earlier phase numbers went

Earlier drafts of the vision used one linear list of phases. Documents written against it use these numbers.

11. Open ordering questions

• Who owns the Go SPIRE node attestor, and where is it released from (A2)?
• Does a service forward the user’s token or exchange it for service-to-service calls (A1, D1)? Both work, and the answer can differ per service.
• Is the operation registry operator-curated, governance-reviewed or service-supplied (C3)? It decides who holds the privilege to widen what every reader may do.
• Which release cycle carries stream B, given that it should carry nothing else?

IAM data plane and check API: design detail

Status: Draft for community discussion

This document holds the design detail that the authentication and authorization vision summarizes in §6.3 and §8.2 to §8.4: how the check API stays honest about the caller’s chain, how it is protected, how replicas stay fresh, and what the data plane is built from. It was moved out of the vision so that the vision can stay a statement of what the proposal is and why. Section numbers written as §N refer to the vision, and link to it. The order in which this is built is in the roadmap. The review that led to the data plane split is in doc/plans/iam-control-data-plane-split.md.

Part 1. The check API (vision §6.3)

Audience: check-API implementers and security reviewers. Skip if you only need the effect on callers; see vision §4.4. Operators sizing a deployment can go straight to the freshness contract and capacity.

The chain facts are not the caller’s to assert

Items 3 and 4 are chain facts in the sense of I1: properties of how the caller authenticated, never of what the caller says. The request shape of §4.4 does not preserve that on its own. An enforcement point parses act, delegation_context and restrictions out of the presented credential, so a data plane that applies items 3 and 4 to the fields it was handed is applying them to data an enforcement point wrote, and a request with delegation_context omitted asks to be decided as though the chain were not delegated. That is the defect class behind OSSA-2026-005 and OSSA-2026-015, relocated from token scope into the check request, where it is harder to see because the field names are the right ones. SVID authentication does not close it: §4.2 is explicit that mTLS identifies the service and never the user it serves, and a compromised service that may assert its own account of a user’s chain can widen every delegation passing through it (§4.8).

So the chain must reach the data plane in a form the data plane verifies for itself, and the proposal takes the direct route: the check request carries the subject’s credential as presented, and the data plane verifies it offline exactly as the enforcement point did, deriving sub, act, delegation_context and restrictions from the verified claims. Parsed fields may still be sent for logging and a cheap pre-flight, but they are advisory — where they disagree with the credential the credential wins, and the disagreement is an audit event. Forwarding discloses nothing new: the callee already holds the credential, and the data plane is internal-only.

Two consequences follow.

• A check whose chain cannot be verified cannot be decided safely. The data plane cannot differentiate an undelegated chain from a delegated one whose context was dropped, and the safe reading of that ambiguity denies everything. A deployment that cannot forward credentials — a legacy Fernet path, or a service in the middle that will not pass one on — therefore runs a declared posture on the same footing as §5.3’s compatibility mode: set per caller, reported as a capability, and understood as a statement that those enforcement points are trusted to assert chain facts. It must not be the default, and a deployment carrying delegated chains should refuse it outright.
• A signed chain assertion is the fallback where forwarding is unacceptable. Keystone can mint at issuance an authenticated, opaque blob binding (sub, act, delegation_context, restrictions, jti) that an enforcement point relays without being able to alter it. It costs a claim and a key, and it says nothing about whether the credential is still live — so it is a fallback, not a preference (§10, question 10).

The request-scoped memo below is keyed by the same argument. Keyed on (subject, target) alone, an answer produced for a subject’s full authority could serve a delegated or restricted call from the same subject on the same target within one request — the escalation above, arriving one lookup later. The key is (subject, chain, target), where the chain component digests exactly the facts items 3 and 4 consume; an undelegated, unrestricted chain digests to a constant, so the ordinary case still shares entries. The same key is mandatory for the opt-in client-side cache described later.

One input deliberately stays outside the evaluation. The data plane knows which service is calling, and uses that for traffic classes and admission control (below), but it does not fold it into the grant decision. A rule such as “only nova-api may call create_port with device_owner=compute:*” (§7.4) is local policy at the callee, where service_credentials is already in the input (§6.2). Making the acting service a dimension of a grant multiplies the grant space by the service catalog, and belongs with resource-level targets (§6.5) if it is taken up at all. The data plane answers “may this subject do this here”, never “may this service ask”.

Answering enough questions discloses the graph

A caller that may ask about any subject, any target and any operation can reconstruct the grant graph a batch at a time. Refusing to ship the graph in bulk while serving an unmetered oracle over the same facts would be a distinction without a difference, so three limits come with the API:

• A caller asks about a subject presenting to it. The forwarding rule above makes this enforceable rather than advisory: the credential is the caller’s proof that the subject is in fact presenting. Asking about a subject whose credential the caller does not hold is a separate, separately granted capability.
• A caller’s target range is bounded where its own reach is. A core service serving arbitrary projects needs no bound; a registered third-party integration is registered against the domains or projects it serves, and a check outside that set is refused rather than answered — and refused, unlike denied, declines to say whether the grant exists.
• Enumeration is measured. Distinct-subject rate and denial rate per caller class belong in the metrics the data plane ships (ADR 0031): a caller walking the graph looks like nothing else on the data plane, many distinct subjects at a high deny rate.

That separate capability is the oracle in administrative clothing, and operators need it: “which of these operations may this user perform on this project” is what an administrator asks while debugging a permission. Answering it is a privileged read of the grant graph, audited as one and granted to tooling rather than to anything on the data path.

This is the new hot path; design it like one

/v4/authz/check replaces token validation on the data path, so its non-functional requirements are not negotiable: batchable, answered from state local to the replica without a per-request round trip to the write path, versioned on terms the response states explicitly, admission-controlled per calling SVID, accounted in decisions rather than requests, and exposed on the internal interface only. Its latency budget is today’s keystonemiddleware cache hit budget, not the miss budget, because it is called on every request (§4.4). The existing assignment-layer caches stay as they are and are not building blocks: ADR 0034 §8 caches the domain-to-driver binding rather than resolved assignments, and ADR 0033 §9 declines to cache OpenFGA results at all. The new work is the in-memory grant index below, which is a replica, not a cache.

Two shapes are explicitly not the answer. A pushed “entitlement snapshot” per subject would recreate the §2.1 frozen-roles problem. Replicating the grant graph to every evaluation point — a bundle each service’s policy engine pulls and evaluates locally — fails for two independent reasons: there is no partition boundary to bound a bundle to, because Nova, Neutron, Cinder and Glance each serve arbitrary projects in arbitrary domains from any node; and an OpenStack grant graph spans mutually distrusting tenants, so “every compute node holds every tenant’s grants” is a disclosure decision, not a caching decision. Both constrain where grant facts come from and say nothing about where policy is evaluated — a per-service OPA sidecar asking this API live is unaffected (§4.4).

Freshness: replicas, not caches

A copy of authorization state has to answer one question: how does it know when it is wrong? There are exactly two answers.

A cache is wrong until someone tells it. Its freshness depends on every write path remembering which entries it affects, and on a lifetime for the cases nobody remembered. Python Keystone’s caching went wrong in precisely this way, repeatedly: a role list, a project’s enabled flag, a token’s validity or a catalog cached under whatever key the reader used; a new write path forgetting the key; region-wide flushes added when targeted invalidation proved incomplete; process-local caches diverging between workers; the lifetime becoming the real safety net; and operators disabling caching to make bugs go away. The refinements do not escape it. A per-subject invalidation counter still has to compute which subjects a write affects — one user added to a large group bumps every member’s counter, a domain-level inherited grant bumps every subject beneath it — which is the same fan-out relocated. A cloud-wide generation number avoids the computation by flushing everything on every write, at the price of a cache that never warms under churn.

A replica is never wrong about what it knows; it can only be behind. The replication protocol maintains its freshness and by construction observes every write, and the replica reports how far behind it is as a number: the Raft applied index against the leader’s commit index, a standby’s replay position against the primary’s, a relation store’s changelog position. No writer has to remember anything. A new grant and a revoked grant propagate at the same speed, which the cache schemes could never deliver for propagation. And a caller that has just written can ask to be answered from a replica that has seen its write, which no cache can offer.

This proposal admits only replicas on the request path. The enforcement point holds no cross-request copy of a decision; the data plane holds a replica of the state and, where useful, an in-memory index derived from that replica and invalidated by the same replication stream. A client-side cache remains possible as an explicit opt-in for deployments that accept a stale window, and it is described last, with the constraints it carries.

Where the replica comes from, per driver

The data plane binary is driver-agnostic in the sense that the same code path handles every store below without a per-driver fork — not in the sense that a keyspace’s driver can differ between the two planes.

Two consequences of that table matter more than its rows.

• The SQL path needs no new component. A database standby is a replica in the sense above, it is operationally familiar, it keeps serving with the primary down, and no driver has to know it is reading one to answer a check. That is what makes the first data-plane deliverable small — but not free: reporting version and honouring min_version is driver work on the SQL path as much as on the Raft one, and §8.3 lists it.
• New authorization state arrives in Raft. §9’s rule that no Python-owned table changes and new state lives in Raft-backed drivers means permissions, permission sets, the operation registry and service grants are in the replicated log from the start, and a learner holds them for free. Assignments themselves are a per-domain choice, and the next subsection compares the two Keystone-owned options.

The replica also removes the objection that sinks every elaborate freshness scheme. No single driver owns all five inputs of a decision (grants, group membership, hierarchy, implied roles, the operation registry), so no composite freshness token over them could be minted consistently, and every write path in four crates would have to remember to bump a counter. A replica mints nothing: within one Raft log every input the log holds is totally ordered, so a decision at applied index N is snapshot-consistent across all of them, and within one database the replay position gives the same guarantee. Only where inputs straddle two stores — SQL assignments beside Raft permissions, an LDAP group beside either — is a decision a join across two versions, and the response says so by reporting both.

A Raft assignment driver, compared with SQL

The existing assignment drivers are SQL (assignment-driver-sql, over the Python-compatible tables) and OpenFGA. A third, Raft-backed driver is the natural companion of the data plane, and worth designing on paper here because its shape decides what a learner can do with it.

It stores a grant as a relation tuple, the way a Zanzibar store does, indexed both ways in the replicated key-value store:

assignment:by_target:{target}:{role}:{actor}  -> { inherited, created_at }
assignment:by_actor:{actor}:{target}:{role}   -> (same value)

where target is system, domain:<id> or project:<id> and actor is user:<id> or group:<id>. Listing is a prefix scan on the matching index, pagination is by key, and a direct check is a point read. Effective resolution walks the same graph the SQL driver walks today — the actor’s groups, the target’s ancestors, then implied roles — and the two inputs that walk needs from other providers, group membership and the project hierarchy, are the crux of the design:

• Mirrored tuples. The driver keeps group:member and project:parent tuples in its own keyspace, written by hooks on the identity and resource providers. Where identity and resource are themselves Raft-backed, the mirror is part of the same log and the graph is consistent at every index — the Zanzibar shape, with userset rewrites expressed as the walk. Where they are SQL, the mirror is a second write and needs the outbox and reconciliation discipline ADR 0035 already specifies for OpenFGA; the driver is then only as fresh as that sync, and the version reports it as a second component.
• No mirror. The driver stores only grants, and the walk asks the identity and resource providers at decision time, as the SQL driver does. On the data plane those providers read their own replicas (a standby for SQL identity), so nothing is lost on freshness; what is lost is single-log snapshot consistency across the join.

Against the SQL driver, benefits and costs are both concrete:

The recommendation follows from the coexistence row. While Python Keystone is still in the picture, assignments stay on the SQL driver and the data plane reads a standby. A domain that only Rust Keystone serves — a greenfield deployment, or one moved under ADR 0034 — can adopt the Raft driver and gains a learner-only data plane and total-order versioning. The two coexist per domain, which is what ADR 0034 was built for, and the data plane serves both from one process.

The freshness contract

The contract is deliberately small, and the same wherever Keystone holds a replica:

• Every response carries version. For a Raft-backed domain the applied log index; for a SQL-backed domain the replay position; for a relation store its changelog position. A decision whose inputs straddle stores reports each component. Enforcement points log it with the decision, which gives the audit trail “allowed against grant state N” (§5.4).
• An input whose store reports no position is named, not hidden. An LDAP directory’s replica is the standing example: it contributes no version component, max_staleness cannot be enforced over it, and the response therefore marks the decision as resting on an input with an unbounded staleness term. An operator who needs a bound cannot get one from that driver, which is half of the capability question §10 asks (question 7).
• A request may carry min_version. A replica behind it waits, up to a bounded time, for its state to reach it; then forwards to a replica that has; then refuses — not denies. The IAM control plane returns the commit version of every grant write, so a client that has just created a grant can ask for a decision that sees it. This is Zanzibar’s consistency token, and Raft provides it natively.
• The operator sets max_staleness. A replica whose measured lag exceeds it stops answering with allows: it refuses, and the caller’s breaker moves to the next replica. This is the freshness dial. Unlike a lifetime it is measured against the write path rather than assumed, and unlike a lifetime it bounds the partition case too.
• Refused and denied are different answers. A refusal says the replica cannot vouch for an answer; a denial says the grant does not exist. Clients treat the first as retryable elsewhere and the second as final, and dashboards count them apart.
• There is no lifetime, no per-result TTL, no mint stamp and no generation number. The response is a fact about state version N. The caller uses it for the request that asked and discards it.
• The data plane’s decision log is complete (§5.4). Every decision reaches a replica, so what the cloud decided is what the replicas logged, with the version each decision was made against.

What the assignment driver still changes is where the replica comes from and whether a version exists at all:

The columns converge wherever the store can report a position, and that is the payoff of not building cache invalidation: one operator-visible freshness model across every driver, instead of a SQL feature beside an external-store limitation.

The in-memory grant index inside the replica

The first data-plane deliverable answers a check by running the same queries the IAM control plane runs, against the local replica, with the request-scoped memo below removing repeats within one batch: a few milliseconds per distinct (subject, target), and enough to ship.

The scaling step is to materialize the grant graph in memory inside each replica — group membership, the hierarchy closure, implied roles, the role-to-permission map — so that a check is a lookup. This is the Zanzibar closure index. What makes it a replica and not a cache is where its invalidation comes from: the Raft apply hook, or logical decoding of the standby’s stream for SQL-backed domains, both of which observe every write. The index carries the version it was built to, the memo is still the only memo, and a replica can disable the index and fall back to queries at any time — so the index is an optimization over the replica, never a second source of truth.

A client-side cache is an opt-in, and it carries its constraints

A deployment may still decide a stale window at the enforcement point is worth a round trip saved — a Python service at the far end of a WAN, or an integration that would rather act on a stale-but-correct decision than fail. That is a client-library option (§9), off by default, and not part of the architecture. Where it is enabled, everything a cache needs applies and none of it is optional: the key includes the chain digest (above); allows are held for a short lifetime and denies may be held longer, because a stale deny is an inconvenience and a stale allow is a security failure; a decision whose chain carries act, delegation_context or restrictions is never served stale; the cached version is logged with the decision, so an auditor can tell “allowed on current data” from “allowed on data N seconds old”; and the lifetime is the bound, since no early-drop signal exists for a cache the write path cannot see. A caller that could ask to be served stale could extend its own revocation window on demand, so the opt-in is a property of the service’s deployment configuration, never of a request.

Two shapes stay refused whatever the freshness mechanism. The pushed “entitlement snapshot” named above stays refused here too, for the same frozen-roles reason. So do source-versioned freshness tokens — sealing the counters of every source a decision consulted into an opaque token, so revalidation is a comparison rather than a traversal: no single driver owns all the inputs, the counter bump becomes a correctness invariant in four write paths with nothing in the type system to enforce it, it accelerates revocation but not propagation, and the counter table becomes the hottest row set in the deployment. A replica gives what that scheme wanted, a version and read-your-writes, without minting anything.

Request-scoped deduplication is the only memo

One inbound API call can evaluate policy many times — the collection-level decision plus one per item, per the re-check requirement below — and those evaluations frequently ask the same (subject, chain, target, operation) question. An in-memory memo scoped to the request and discarded with it removes the repeats, and has no freshness problem to solve because it lives and dies with a single request. It also gives a request one consistent view: a request asking the same question twice should not get two different answers because an assignment changed in between, which is why the data plane answers a batch against one version too.

This is the caller-side counterpart of ADR 0030, which solves the same problem inside Keystone. Nothing in oslo.policy provides it today, so it is part of the check-type work in §9.

Capacity is isolated per caller class, not merely rate-limited

Rate limiting caps what one caller can do to itself. It does nothing about the problem this API creates at scale: an unrelated integration’s traffic degrading queueing and latency for the core services sharing the same data plane. Nova’s checks are not optional — one that does not complete is a user request that fails — while a third-party reporting integration’s checks are sheddable, and the two must not compete for the same queue on equal terms.

So the data plane declares traffic classes with per-class concurrency limits and admission control, shedding the lower class first, with separate deployments as an option on top rather than as the mechanism. The isolation key costs nothing: the data plane already authenticates every caller by SVID on the internal interface, so the SPIFFE trust domain or path prefix is available without inventing an identity concept for it.

Two things about the cost model:

• Accounting is in decisions, not requests. The API is deliberately batchable, so one request can carry a thousand (target, operations) pairs, and a per-request limit is bypassed by batching. ADR 0022’s rate limiting is request-count limiting keyed on IP or username with no notion of per-request cost and no concurrency control, so this is new work rather than an existing mechanism to configure.
• The number to size is decisions per second, not replica count. The per-item re-check below amplifies one user action into a page’s worth of decisions, so a deployment guide that documents only replica placement (§8) has documented the smaller half of the capacity question.

Both depend on being able to see the traffic, which makes per-trust-domain attribution a prerequisite and not a later nicety: decisions per second, replica lag, refusal rate, denial rate and shed count broken down by caller class belong in the metrics (ADR 0031) that ship with the data plane. Quotas that cannot be measured cannot be set.

Collection reads and the per-item re-check

The security model requires list endpoints to re-enforce the per-item read policy against each record’s own identifiers (I8), and forbids replacing that with a permissive collection filter (I8a). A cross-project list under system-scope="all" therefore cannot be one decision: it is a collection-level decision plus one per-item decision per record. Three things keep that from being a per-record round trip:

• Batch checks. The API takes a list of (target, operations) pairs in one call, so a page of results is one request, not one per row.
• Grant-shaped short-circuit. When the collection-level decision was allowed through a system or domain grant that covers every project in the page, the per-item check is answered from that same grant rather than re-derived. This is a short-circuit in the evaluation, not a relaxation of I8: each item is still decided against its own owner, and an item owned outside the grant’s reach still has to be decided on its own and dropped if denied.
• The request-scoped memo. Decisions within the batch are memoized per (subject, chain, target), so a page whose rows share few owners costs few distinct decisions, and the whole batch is answered against one version.

The consequence is that cross-project listing is more expensive under this model than a system-scoped token that services interpreted themselves, and deployments must size the check API for it.

Bootstrapping and Keystone’s own authorization

Keystone is itself a service with authorized APIs. It evaluates its own policies against its own assignment data in-process — it does not call /v4/authz/check over the network to authorize a call to itself — and /v4/authz/check is itself authorized by the caller’s SVID at the internal interface, not by a recursive check. The first grant in a new deployment comes from the existing bootstrap path (keystone-manage, operating on the store directly), exactly as today. In addition, Keystone exposes a local admin interface over a Unix domain socket: it carries the full API, is reachable only from the host, and authenticates the caller by SVID, so a high-security perimeter can drive administrative calls without a bearer token and with the same policy checks the network API applies.

Part 2. The data plane (vision §8.2 to §8.4)

Audience: storage and Raft maintainers, and implementers of keystone-dataplane. Skip if you only need the outage cost; see vision §8.5.

Vision §8.2: What the data plane is, concretely

In the Rust implementation the data plane is a second binary target in the keystone crate, keystone-dataplane, built from the same library. The two binaries are built in separate invocations, and the data-plane build never names the write-side router module, so the absence of a write path is a property a reviewer checks in the build rather than a flag they trust at runtime:

• Routes. /v4/authz/check; JWKS, .well-known and the revocation feed; JWT validation against that JWKS; the catalog reads services and SDKs need on the request path. No token issuance, no login, no refresh, no attestation, no grant or lifecycle writes, no configuration, no SCIM. Legacy Fernet validation (GET /v3/auth/tokens, Mode B) stays on the IAM control plane deliberately: Fernet keys are a symmetric repository, so a replica able to validate a Fernet token is a replica able to mint one, and putting that repository in every failure domain would undo §8.4 before it starts. The legacy path therefore keeps the legacy availability, which is one more reason for a deployment adopting the data plane to finish the move to JWT.
• Drivers. The same drivers as keystone, each pointed at a replica (§6.3 table). The SQL connection is to a standby, which the database makes read-only, and the database user has SELECT only, so a misconfiguration pointing at the primary still cannot write. Mutating provider methods are not compiled in; where a shared path could reach one — a hook, an audit side effect — the execution context carries a read-only marker that makes the call fail loudly.
• Storage role. The replica joins the Raft cluster as a permanent learner under a distinct SPIFFE role (spiffe://<td>/keystone/storage/learner). The leader’s gRPC interceptor, which already parses the role segment and gates operator RPCs on it, rejects client_write, change_membership and every operator RPC from that role. A learner never promotes itself and cannot be promoted by mistake: the role decides, not the membership.
• Configuration. A [dataplane] section with max_staleness, the bounded wait for min_version, and the peer replicas to seed from; the [distributed_storage] section gains role = "learner".

flowchart TB
    subgraph cp["IAM control plane: writes and authentication"]
        k1["keystone (voter)"]
        k2["keystone (voter)"]
        k3["keystone (voter)"]
        pg["PostgreSQL primary"]
        k1 --- pg
        k2 --- pg
        k3 --- pg
    end
    subgraph r1["Region A data plane (optional)"]
        d1["keystone-dataplane<br/>learner + PG standby"]
        d2["keystone-dataplane<br/>learner + PG standby"]
        d3["keystone-dataplane<br/>learner + PG standby"]
        s1["nova-api, neutron, ...<br/>oslo.policy: memo only, no cache"]
        s1 -->|"check, batched, version"| d1
        s1 -->|"failover"| d2
        s1 -->|"failover"| d3
    end
    subgraph r0["Small cloud"]
        s0["services"]
    end
    s0 -->|"check, directly"| cp
    cp -.->|"Raft log to learners,<br/>WAL streaming to standbys"| r1

Vision §8.3: What the storage and SQL layers have to gain first

The Raft storage exists, and the data plane leans on it, but five things are missing between “a learner” and “a replica that serves”, and they are the concrete content of the static-stability requirement:

1. Local reads on learners, with the version. Today every read on a non-leader node forwards to the leader after a ReadIndex round trip, and reads locally only as a fallback when the leader is unreachable. A node in the learner role serves reads from its own state machine without ReadIndex and returns the applied index with every result. This needs an amendment to ADR 0016-v2 §3, and this document does not make one. That section puts group membership in Tier 2 precisely because a stale membership read can allow a member who was just removed, and the argument offered here — that a learner is a replica by definition, so reporting its lag and refusing past max_staleness is more honest than hiding the staleness behind a forward to the leader — is an argument for amending it, not a substitute for doing so. The tier rule stands as written for voters, where a read is a live administrative fact. local_reads_mode, which the storage documentation already describes, is where the learner behaviour would live.
2. Lag as a first-class metric. Leader commit index, learner applied index and their difference, exported by both sides (ADR 0031), with the max_staleness refusal counted separately from denies.
3. Seeding without the leader. Snapshot install is leader-driven today, so a replacement replica cannot cold-boot during an outage of the write path. Either the snapshot transfer is generalized to any peer that holds a newer snapshot, over SVID mTLS, or the documented procedure is keystone-manage storage restore from the latest backup followed by catch-up. The second exists and is enough to start; the first is what makes a replacement replica routine rather than a runbook.
4. A learner SPIFFE role, as above: a small addition to the existing role parsing and interceptor.
5. Version and lag on the SQL path. the SQL driver still has to report the standby’s replay position as version, export its lag against the primary as a metric, and the write path still has to return the commit LSN of a grant write so that min_version means anything on a SQL-backed domain. Small work, but the freshness contract is not met without it.

Vision §8.4: Keeping secrets away from replicas

A learner applies every log entry, and applying an encrypted entry needs the data encryption key, so a learner today decrypts everything the log holds: OAuth2 private signing keys, refresh-token families, API-key hashes and whatever else a Raft-backed driver stores. More replicas in more places means more copies of that material — the one place where the data plane widens the attack surface rather than narrowing it. Two designs close it, adoptable in order.

• Tier-keyed encryption. ADR 0016-v2 already binds a sensitivity tier into every record’s authenticated data and derives purpose-specific sub-keys from the master key. The extension is a second key hierarchy for Tier 2 and Tier 3 payloads: the writer encrypts the payload with a tier key before proposing it, under the existing log and state encryption, and only voters hold that tier key. A learner applies the entry, stores the inner ciphertext, and answers a read of such a key with “not readable in this role”. The change is confined to the storage crate’s encrypt and decrypt paths and to KEK provisioning; learners still carry the ciphertext volume, and a tier-key rotation becomes a voter-side re-encryption sweep.
• A second Raft group. Secret material — sessions and refresh families, API-key hashes, credential plaintexts, private signing keys — moves to a group the data plane is not a member of, while authorization state stays in the group it follows. The storage crate is already generic over keyspaces, so two storage instances with separate membership and paths are configuration plus a keyspace-to-group routing table. The cost is two clusters to operate and the rule that no write may span groups. It is the cleaner boundary, and the one that stops learners carrying secret ciphertext at all.

Tier-keyed encryption is enough for replicas inside the control-plane trust zone; a replica in a remote region, or on a host shared with other services, is the case for the second group. Which secrets belong in which tier is the storage ADR’s decision, not this document’s (§10, question 13).

A Public Key Management Service for OpenStack: Architecture

Status: Draft for community discussion

Date: 2026-09-13

This document specifies the architecture of a new project: a public Key Management Service (KMS) for OpenStack-based clouds. It is a companion to the authentication and authorization vision and assumes that vision’s building blocks — SPIFFE/SPIRE workload identity, scope-less tokens with a per-request target, a Keystone-backed authorization decision, OPA policy, CADF audit, handler-level rate limiting — are available, because the KMS is built on keystone-rs primitives and reuses its crates wherever the shape fits.

The service is deliberately not a rewrite of Barbican. Barbican is a secret store with a plugin backend; what operators of a public cloud need is a managed key management service with the operational profile of AWS KMS or GCP Cloud KMS: a hard multi-tenant blast-radius story, an envelope-encryption data path that survives the loss of any single backend, standard interfaces (KMIP, the OpenBao/Vault API) so that existing customer software works unmodified, and a clear separation between what a tenant may reach from the internet and what only the cloud’s own services may reach.

Under the hood the service does not implement its own cryptographic storage engine. It operates a fleet of OpenBao clusters, shards OpenStack tenants across them for failure isolation and scalability, and puts a Rust control and data plane in front of that fleet. The fleet is an implementation detail of the service, never a contract with the tenant — not even on the surface that speaks the OpenBao API, which the KMS implements itself rather than proxying (§5.6).

Every section states its own requirements and limits. Where more than one design is defensible the options are presented with a recommendation, because this is a document to argue about, not a finished decision record. Section 19 gives the phased plan; the first useful deliverable is narrow on purpose.

1. The vision in one page

A public cloud cannot ask its customers to trust a key management service that is a thin wrapper over a single database. It has to answer four questions concretely:

1. Where does my key material live, and who can reach it? Answer: inside a region, inside one OpenBao cluster shard dedicated to a small set of tenants, behind a barrier key that is itself sealed by a central seal authority or an HSM. No operator path reads plaintext key material; no tenant network path reaches the storage layer at all.
2. What happens when something breaks? Answer: the blast radius of a shard failure is the tenants on that shard, not the region. Envelope-encryption operations — the hot path, and the one that users notice — are served by a stateless crypto tier that keeps working for the lifetime of its cached wrapped keys even when the shard behind it is unavailable.
3. How do I integrate what I already run? Answer: three interfaces. A native REST API, KMIP for storage arrays, databases and backup software, and an OpenBao/Vault-compatible surface (transit, kv v2, and the minimum of sys) so that software written against Vault works without change — most immediately, so that a customer’s own OpenBao can auto-unseal against the cloud’s KMS.
4. How do my workloads authenticate? Answer: with SPIFFE. Customer workloads are assumed to have SPIRE support — either the cloud’s own SPIRE (VMs attested per ADR 0032, cluster nodes via the same path) or a customer-run SPIRE federated with it. Long-lived secrets in guest images are a legacy path, not the design centre.

The shape that follows from those answers:

and one rule that runs through the whole design:

The tenant-facing plane never talks to the storage plane, and the storage plane never terminates a tenant connection. Every crossing is mediated by a service that holds a SPIFFE identity, is authorized per request, and is audited.

2. Scope, non-goals, and the relationship to what exists

2.1 In scope

• Symmetric key management: create, rotate, enable/disable, schedule deletion, encrypt/decrypt of small payloads, envelope-encryption data-key generation.
• Asymmetric key management: sign/verify, wrap/unwrap, public-key retrieval.
• Secret storage (opaque blobs) for the Barbican and kv use cases.
• Customer-managed keys (CMK) for OpenStack services: Cinder volume encryption, Glance image encryption, Swift/S3 object encryption, Manila, database and backup encryption.
• Bring-your-own-key import and, where policy allows, export under wrap.
• KMIP server for third-party clients (storage arrays, Oracle/MSSQL TDE, backup software, vSphere).
• OpenBao/Vault API compatibility for transit and kv v2, sufficient for a customer OpenBao to auto-unseal and for Vault-aware software to work.
• Regional HA, multi-region key policy, and disaster recovery.

2.2 Out of scope (at least initially)

• Being a general-purpose secret manager with dynamic secrets engines (databases, PKI, cloud credentials). Those are what a customer’s own OpenBao is for; the cloud KMS exists partly to make running one safe.
• Certificate authority services. A CA is a distinct product with distinct compliance; the KMS may hold the CA’s private key, and that is the boundary.
• Replacing SPIRE’s own CA or Keystone’s signing keys. Those have their own bootstrap chain and adopting the KMS for them is a later, separate decision (§9.6).
• Confidential computing attestation-bound release of keys. It is an authorization condition on top of this architecture rather than a change to it, and §2.6 states what the service would and would not own if it were added.

2.3 Relationship to Barbican

Barbican defines the OpenStack contract that Cinder, Glance, Nova, Octavia and Manila already speak (castellan). The KMS therefore ships a Barbican v1 compatibility surface (§5.8) so existing services keep working from day one, and offers the native API for everything Barbican cannot express (grants, envelope operations, KMIP, key policies, multi-region behaviour). Barbican’s own plugin model is not reused: a plugin inside Barbican’s process cannot give the blast-radius or network-isolation properties of §1.

2.4 Relationship to keystone-rs

The KMS is a separate project with its own repository, release cadence and database. It depends on Keystone for identity and authorization decisions, and it borrows Keystone’s engineering: axum handlers, SeaORM persistence, the OPA integration (ADR 0002), the security context (ADR 0017), handler-level rate limiting (ADR 0022), CADF audit (ADR 0023), Prometheus metrics (ADR 0031), pagination (ADR 0029) and the per-request cache (ADR 0030). Shared code is extracted into crates published from keystone-rs rather than vendored, so that a fix to the audit pipeline or the rate limiter benefits both. Keystone’s PKCS#11/TPM KEK work (doc/plans/0016-v2-pkcs11-tpm-kek.md) is the direct ancestor of the HSM integration in §10.

The dependency on Keystone is a runtime dependency with the same posture as the one in vision §8: authentication is validated offline, authorization is a cached live decision, and the KMS fails closed when no cached decision applies. Because a KMS outage is a data-availability outage for everything encrypted with it, §11.4 imposes tighter requirements on that cache than a compute API would need.

2.5 Relationship to OpenBao

OpenBao is a premise of this design, and a premise deserves an argument. Three answers to “under whose barrier does key material live” were considered:

The decisive argument is that third option’s failure, and it is not the one that first suggests itself. Writing a barrier is expensive, but the sharper reason is this: §10 promises that an HSM “is never required for the service to function”, and §9’s seal cluster — itself an OpenBao with a fleet-unseal transit mount — is the software root of trust that makes the promise keepable. Remove OpenBao and a deployment must either make an HSM mandatory, pricing out every operator who does not already have one, or grow its own software seal, which is rebuilding the least forgiving part of OpenBao with none of its review history. OpenBao is what lets the HSM stay optional, and that is a better justification than not wanting to write a barrier.

The secondary argument is the compliance narrative of §20. “The key material is inside an OpenBao barrier” is a sentence an auditor already accepts; “the key material is inside our barrier” is one the project must prove from first principles, every time, for every scheme.

What OpenBao is not asked to be is the region’s cryptographic engine. §7 makes mode B the default, under which the hot path reaches a shard roughly once per key per cache lifetime, and §6.3 is explicit that transit’s non-exportability is precisely why the offload cannot be built on transit alone. The dependency is therefore deliberately narrow, and §8.6 writes down its exact surface so that it stays narrow.

2.6 Relationship to confidential computing

Two different things travel under the name “key broker”, and separating them is most of the analysis: one is functionality the service would sell, the other is a way the service could run.

What the end user wants. A tenant running a confidential VM or a confidential container wants a key released only to a workload whose measurement it recognises. The shape the ecosystem has settled on is two components: a key broker that runs an attestation handshake with the workload and hands back a resource, and an attestation service that appraises the evidence against reference values and hardware endorsements. The KMS is the right holder of the release decision — it already holds the key, it already evaluates a per-object policy in OPA (§13.3), and a broker able to release keys the KMS custodies without the KMS deciding would split exactly the authority §13 exists to consolidate. The KMS is the wrong owner of the appraisal: maintaining reference values, vendor endorsement chains and TCB levels across TEE generations is a distinct product with a distinct compliance lifecycle, which is the same reason §2.2 excludes certificate authority services.

The split that follows:

What the service itself needs: nothing. Attestation is also a candidate root of trust for the KMS’s own processes — a seal cluster that releases a shard’s unseal key only to a node that has proved its measurement (§9.3), or a kms-crypto running inside a TEE so that the plaintext KEK cache of §7.3 sits in memory the host cannot read. Both are attractive and neither may become required: principle 6 makes the HSM optional, and a service that could not start without attested hardware would be that promise broken in the other direction. On the service’s own path, attestation is therefore an additional Tier-0 option ranked alongside TPM and HSM sealing, never a prerequisite. The distinction also decides when each may be considered: a tenant-facing broker is a feature that can wait for demand, whereas anything the service needs in order to start has to be settled before Phase 1.

Adding the broker is cheaper than it first looks, because each piece is already present in some other guise:

• The release primitive exists. Fetching a brokered resource is a read of an opaque secret (§5.4); releasing a key is unwrap or data-key.
• Wrapping the released material to a public key the caller has proved it holds is the BYOK import-token machinery of §5.4 run backwards, with the ephemeral public key taken from the evidence instead of minted by the KMS.
• The AAD of §6.4 already carries caller_scope_digest; a digest of the appraised claims goes there with no format change. That buys a property worth having on its own — a DEK minted for an attested workload cannot be replayed into an unattested context, even by the same project.
• Appraisal policy and KMS policy are both Rego, evaluated by the engine the project already runs.

What is not cheap is precisely what the recommendation above declines to own. Appraisal needs endorsement material fetched from a hardware vendor outside the region, which principle 7 does not forbid but does not welcome. A firmware update that moves a measurement is a fleet-wide availability event with a cause the KMS cannot see coming. And reference values describe the tenant’s image, so the tenant owns them — a tenant who gets them wrong loses access to its own data with nothing broken to repair, which is a support burden of a kind the rest of this design works hard to avoid.

Three consequences are carried into the rest of the document: an identity track that is not really a new one (§4.4), a condition on grants (§13.4), and a freshness rule the caches must not quietly undo (§7.3). None of them is a Phase 1–5 concern; §21, question 12 says what would make it one.

3. Principles

1. Blast radius is a design parameter, not an outcome. Every tenant is assigned to a shard; the shard is the unit of failure, upgrade, capacity and incident scope. A shard is sized so that its loss is an incident, not a catastrophe.
2. Plaintext key material exists in exactly two places: inside an OpenBao barrier (or HSM), and transiently in the memory of a crypto-tier process that is currently performing an operation. It is never written to the KMS’s own database, never logged, never returned by an API except where the tenant explicitly asked for a data key or an authorized export.
3. The hot path must not be the fragile path. Envelope operations are orders of magnitude more frequent than key-management operations and are served by a tier that can be scaled and degraded independently of the backend fleet.
4. Internal and external are different services that happen to share code. They bind different addresses, on different networks, with different authentication, different policy, different rate limits and different audit streams. A handler is reachable from exactly one of them unless a deliberate decision says otherwise.
5. Standards where customers already stand. KMIP and the Vault API are not nice-to-haves; they are the reason an enterprise can adopt the service without rewriting software.
6. The HSM is an option, never a prerequisite. A deployment without an HSM must be fully functional and honestly documented as to what it does and does not guarantee. A deployment with an HSM must not have to re-architect.
7. No dependency on any external public KMS. The service is the root of trust for the cloud it serves. Everything it needs to bootstrap must be derivable from operator-held material, a local HSM, or its own seal cluster.
8. Region is a hard boundary by default. Key material does not leave its region unless the key was explicitly created as a multi-region key and the tenant asked for it.
9. Crypto-agility from the first line. Algorithms, key sizes and wrapping formats are versioned in the wire format and in storage, so that post-quantum wrapping (ML-KEM) and signing (ML-DSA) are additions, not migrations.
10. Everything is recorded. Every administrative action and every fleet operation emits its own CADF event. Every key use is recorded too, but at data-plane volumes the unit of record is allowed to be an aggregate (§14): what may never be lost is the identity, the key version and the outcome of any failed, denied, key-material or management operation. A KMS that cannot prove what it did is not usable for compliance; a KMS that emits one event per object write is not affordable, and the design says which is which rather than pretending the tension does not exist.

4. Target architecture

4.1 Component overview

flowchart TB
    subgraph ext["External zone (tenant / internet)"]
        app["Tenant application<br/>SPIFFE SVID"]
        bao["Tenant OpenBao<br/>auto-unseal client"]
        kmipc["KMIP client<br/>storage array, TDE, backup"]
        cli["Human / CLI / Horizon"]
    end

    subgraph edge["Edge tier (kms-api)"]
        rest["v1 REST API"]
        vault["Vault-compatible surface<br/>transit, kv v2, sys subset"]
        kmip["KMIP front end<br/>TTLV over TLS"]
    end

    subgraph crypto["Crypto tier (kms-crypto)"]
        env["Envelope engine<br/>KEK cache"]
        pkcs["PKCS#11 client<br/>(optional)"]
    end

    subgraph ctl["Control plane (kms-control)"]
        meta["Key metadata, grants,<br/>lifecycle, rotation"]
        place["Placement and<br/>fleet orchestration"]
    end

    subgraph fleet["Backend fleet (per region)"]
        s1["Shard A<br/>OpenBao cluster"]
        s2["Shard B<br/>OpenBao cluster"]
        s3["Shard C<br/>OpenBao cluster"]
    end

    subgraph rot["Root of trust"]
        seal["Seal cluster<br/>central OpenBao"]
        hsm["HSM partitions<br/>(optional)"]
    end

    ks["Keystone<br/>authn / authz"]
    spire["SPIRE"]
    opa["OPA"]

    app --> rest
    bao --> vault
    kmipc --> kmip
    cli --> rest
    edge --> crypto
    edge --> ctl
    crypto --> fleet
    crypto -.-> pkcs
    pkcs -.-> hsm
    ctl --> fleet
    ctl --> place
    fleet -->|"unseal"| seal
    fleet -.->|"unseal (alt)"| hsm
    seal -.->|"unseal"| hsm
    edge -->|"check(sub, op, target)"| ks
    edge --> opa
    spire -.->|"SVIDs"| edge
    spire -.->|"SVIDs"| crypto
    spire -.->|"SVIDs"| ctl
    spire -.->|"SVIDs"| app

4.2 The five processes

One piece of shared infrastructure is load-bearing enough to name here rather than leave implied: the invalidation bus, the path by which kms-control tells every kms-crypto replica that a key has been disabled, a grant revoked or a deletion scheduled (§7.3). It is what makes those operations take effect in seconds rather than at the end of a cache TTL, so it is on the critical path for a security property, not for throughput. It is regional, it carries no key material (only key identifiers and a monotonic epoch), and its failure mode is explicit: a kms-crypto replica that loses the bus for longer than one TTL stops admitting new cache entries and drains, rather than serving from a cache it can no longer trust. Implementation options — a message broker, a Raft watch on the control plane, or direct fan-out over the internal mTLS mesh with kms-control holding the replica list — trade operational surface against delivery guarantees; direct fan-out is the recommendation, because the replica count is small, the fencing decision has to be synchronous anyway, and it adds no new system to the trust boundary.

kms-api and kms-crypto are separate processes even in a small deployment. That split is what makes the network isolation of §12 expressible, and it is what lets the crypto tier be given a different hardening profile (no inbound tenant traffic, memory locking, core dumps disabled, optional HSM session ownership).

4.3 Deployment topology

Per region:

• ≥3 kms-api replicas behind the public and internal load balancers, in distinct availability zones.
• ≥3 kms-crypto replicas, AZ-distributed, no tenant-facing address.
• ≥2 kms-control replicas; schedulers use a lease.
• N shards; each shard is an OpenBao cluster of 3 or 5 nodes spread across AZs, with integrated (Raft) storage.
• One seal cluster: a small, dedicated OpenBao cluster (3 or 5 nodes) used only as a transit auto-unseal authority for the shards (§9).
• Optional HSM partitions, at least two, in different failure domains.

Per cloud (across regions): the control plane’s metadata is regional. There is no global write path. Cross-region concerns — multi-region keys, global key identifiers, replication policy — are handled by explicit, auditable replication, never by a globally shared database (§11).

4.4 Identity tracks

The last two rows are the important ones. When Cinder attaches an encrypted volume it is acting for a user; per ADR 0036 the delegation is a fact of the authentication chain, and the KMS policy keys on input.credentials.delegated_project_id, never on a scope the caller supplied. A service that has an SVID but no delegation may not unwrap a tenant’s key; this is precisely the invariant the security model already states, applied to the one service where violating it is catastrophic. The confidential-computing row is deliberately not a new mechanism. The appraisal happens before the KMS sees the caller — at SVID issuance, or in a verifier whose signed result the caller presents — so what reaches policy is the principal the KMS already understands, with claims attached (§2.6).

5. API surfaces and the external/internal split

5.1 Zones

Four zones, with distinct listeners, certificates, policy bundles and audit streams. “Zone” here is an architectural term; §12 maps it onto networks.

The same handler is never mounted in more than one zone by accident: the router is built per zone from an explicit list, and a test asserts that the public router does not contain any handler tagged internal or management. That test is the enforcement mechanism; a code review is not.

5.2 What is public and what is not

¹ Only for keys created with exportable = true, only under an approved wrapping key, and only when policy allows; the flag is immutable after creation and is recorded in every audit event for that key.

² The fleet unseal path is not an API the KMS serves in any zone — it is a shard node speaking to the seal cluster over the seal network (§9), and the row exists to say so. What the management zone offers is its configuration (which shard has which unseal key, and rotating them), which is the “seal rotation” row below.

³ Break-glass only, through kms-manage, with quorum approval and an audit record, along the lines of ADR 0028.

5.3 Three surfaces, three base paths

The native API, the Vault-compatible surface and the Barbican-compatible surface each want to own /v1/, and two of them want /v1/secret(s). That collision has to be resolved in the routing layer before any of the three is written, and the resolution is visible to clients, so it belongs here rather than in an implementation note.

Distinct hostnames are recommended. They keep every compatibility surface byte-identical to the API it imitates, they let the KMIP and Vault surfaces be withdrawn from a deployment without touching the native one, and they give the network layer (§12) a natural place to apply different exposure per surface. Distinct base paths are a workable fallback where certificate management is the binding constraint — the Vault client preserves the path component of its configured address, so address = "https://kms.<region>.<cloud>/vault" reaches /vault/v1/transit/… — but it has to be verified per client rather than assumed, and KMIP has no equivalent (it is a separate port regardless).

The catalog entries are distinct service types (key-manager for the Barbican surface, so that castellan finds it unchanged), which also means an operator can decline to register a surface they do not want to offer.

5.4 Native v1 REST API

Resource shapes, with the OpenStack conventions of the v4 Keystone API (ADR 0004) — cursor pagination (ADR 0029), problem+json errors, OpenStack-Target for scope where the target is not implied by the addressed resource.

POST   /v1/keys                       create a key
GET    /v1/keys                       list keys in the target project
GET    /v1/keys/{key_id}              describe (metadata only, never material)
PATCH  /v1/keys/{key_id}              enable, disable, rename, retag
DELETE /v1/keys/{key_id}              schedule deletion (waiting period)
POST   /v1/keys/{key_id}/cancel-deletion
POST   /v1/keys/{key_id}/rotate       new version, old versions kept
GET    /v1/keys/{key_id}/versions
POST   /v1/keys/{key_id}/import       BYOK, material wrapped to an import token
GET    /v1/keys/{key_id}/import-token get wrapping key + nonce for BYOK
GET    /v1/keys/{key_id}/public       public key (asymmetric only)

POST   /v1/keys/{key_id}/encrypt      small payload, <= 8 KiB
POST   /v1/keys/{key_id}/decrypt
POST   /v1/keys/{key_id}/rewrap       re-encrypt ciphertext to latest version
POST   /v1/keys/{key_id}/sign
POST   /v1/keys/{key_id}/verify
POST   /v1/keys/{key_id}/data-key     envelope: plaintext + wrapped DEK
POST   /v1/keys/{key_id}/data-key/wrapped-only
POST   /v1/keys/{key_id}/unwrap       wrapped DEK -> plaintext DEK
POST   /v1/keys/{key_id}/hmac
POST   /v1/random                     randomness from the region's CSPRNG

POST   /v1/keys/{key_id}/grants       delegate specific operations
GET    /v1/keys/{key_id}/grants
DELETE /v1/keys/{key_id}/grants/{grant_id}

POST   /v1/secrets                    opaque secret store (Barbican-shaped)
GET    /v1/secrets/{secret_id}

GET    /v1/quotas                     per-project limits and usage
GET    /v1/usage                      current billing period, per key and class
GET    /v1/regions                    regions this deployment can replicate to
GET    /v1/keys/{key_id}/regions      where this key exists today

Batch variants of encrypt, decrypt and data-key exist from the start. A storage service encrypting a thousand objects should make one request, and the per-item results carry per-item errors. Batching is the single most effective lever on backend load (§7).

5.5 KMIP

KMIP is how the enterprise storage world talks to a KMS, and shipping it is a large part of why this service exists rather than a Barbican plugin.

• Protocol versions: KMIP 1.4 and 2.1, TTLV encoding over TLS. JSON and XML encodings are deferred; almost nothing in the field uses them.
• Profiles: target the KMIP 2.1 Baseline Server profile plus Symmetric Key Lifecycle, Basic Cryptographic and Opaque Managed Object server conformance, and their 1.4-era equivalents where the profile names differ. Between them these cover the common clients (VMware, NetApp, Dell, Oracle TDE, Veeam, Commvault) — but “covers” is a claim to be proven by the interoperability matrix of Phase 5, not by reading the specification.
• Operations: Create, CreateKeyPair, Register, Get, GetAttributes, GetAttributeList, AddAttribute, ModifyAttribute, DeleteAttribute, Activate, Revoke, Destroy, Locate, Check, Query, DiscoverVersions, Encrypt, Decrypt, Sign, SignatureVerify, MAC, RNG Retrieve. Get with a KeyWrappingSpecification is how KMIP clients do envelope encryption, and the KMS maps it onto the same data-key path as §5.4.
• Identity: mTLS client certificate only. Two mappings are supported: a SPIFFE ID in the SAN (preferred; the client is a workload with an SVID), or a certificate issued by a tenant-registered CA with a subject pattern the tenant mapped to a principal. The second exists because a storage array cannot run a SPIRE agent. Certificate-to-principal mapping reuses the mapping engine (ADR 0020) — the same rule language that maps an OIDC assertion maps a certificate’s subject.
• Tenancy: a KMIP connection is bound to exactly one project, derived from the client identity at handshake time. Locate never crosses that boundary. There is no KMIP operation that can name another tenant’s object, and object identifiers are opaque per-tenant handles, not global key IDs.

Implementation options.

Option A with the codec in a separate crate, no_std-friendly and fuzzed (the repository already runs cargo-fuzz), is the recommendation. TTLV parsing is a hostile-input surface reached over the network and deserves that treatment.

5.6 The OpenBao/Vault-compatible surface

A deliberate, bounded subset — presented as an API the KMS implements, not as a transparent pass-through to a backend cluster. Requests are parsed, authorized, rate-limited, audited, and then executed against the shard. The tenant never holds an OpenBao token for a fleet cluster.

The auto-unseal use case needs only four of these (transit/encrypt, transit/decrypt, auth/*/login, sys/health), which is why it is Phase 1 (§19). A customer configures their own OpenBao with:

seal "transit" {
  address         = "https://kms.<region>.<cloud>/"
  mount_path      = "transit/"
  key_name        = "unseal-prod"
  tls_ca_cert     = "/etc/ssl/cloud-ca.pem"
  tls_client_cert = "/run/spire/certs/svid.pem"
  tls_client_key  = "/run/spire/certs/svid.key"
}

and never handles an unseal key again. Because customer workloads are assumed to have SPIRE support, the client certificate is an SVID and the credential is rotated by the SPIRE agent; no secret is written into the customer’s config management.

One detail decides whether that last sentence is true. The upstream transit seal stanza carries a token field and treats it as required, because it was written against a Vault that authenticates with tokens rather than with client certificates. The KMS is not a Vault: it authenticates the mTLS peer, and the token is advisory. Three ways to close the gap, in preference order:

1. Accept the SVID and ignore the token. The seal stanza carries a placeholder; authorization comes entirely from the client certificate. This is the recommended path and is why tls_client_cert is the load-bearing line in the configuration above.
2. Issue a token to a certificate. A short-lived, single-key token from auth/cert/login, renewed by a sidecar that writes it where the seal stanza reads it. Needed if the client refuses to start without a syntactically valid token.
3. A Keystone application credential scoped to exactly one transit key, for a customer whose OpenBao predates their SPIRE rollout.

The KMS records which of the three was used on every unseal, and an operator can require (1) per project so that a fallback cannot be silently adopted.

Design decision — synthesize, do not proxy. A transparent reverse proxy to a shard would be less code and is tempting. It is rejected: it leaks backend identity and version, makes the tenant’s blast radius the cluster’s blast radius, defeats per-request authorization against Keystone, and makes shard migration a customer-visible event. The compatibility surface is an implementation of the Vault wire contract, and that contract is version-pinned and covered by contract tests against a real OpenBao client.

5.7 Internal service API

The internal zone carries what OpenStack services need and tenants must not reach:

• Envelope operations at scale, batched, with service-level rate limits and a distinct quota pool, so that a Cinder mass-attach cannot starve tenant API traffic.
• The delegated-unwrap path: “unwrap this DEK for volume V of project P, on behalf of user U”, authorized against the user’s grant on the key, with the acting service recorded (act).
• Key-usage attestation: a service can ask “is key K usable right now for project P”, which is what a scheduler needs before it places a workload that will require the key.
• Bulk re-wrap notification: when a tenant rotates a CMK, services that hold ciphertext under the old version subscribe to an event stream and re-wrap lazily.

5.8 Barbican compatibility

A Barbican-shaped surface (secrets, containers, orders, ACLs, consumers) on its own hostname or base path per §5.3, sufficient for castellan and the existing OpenStack integrations. It is a translation layer over the native model: a Barbican “secret” is a KMS secret or a key, a Barbican ACL becomes a KMS grant, an “order” for a symmetric key becomes a key creation. Not everything maps — Barbican’s certificate orders do not, and return 501. Compatibility is scoped by an explicit conformance test suite run against castellan’s own tests, not by best effort.

6. Key hierarchy and cryptographic model

6.1 Hierarchy

flowchart TD
    hsm["HSM root key<br/>(optional, non-exportable)"]
    sealkey["Seal-cluster barrier key"]
    sealtransit["Per-shard unseal key<br/>(transit key in seal cluster)"]
    barrier["Shard barrier key<br/>(OpenBao master key)"]
    cmk["Customer master key (CMK)<br/>versioned, per project"]
    dek["Data encryption key (DEK)<br/>ephemeral, per object/volume"]
    data["Tenant data"]

    hsm -->|"seals"| sealkey
    sealkey --> sealtransit
    sealtransit -->|"wraps"| barrier
    barrier -->|"protects storage of"| cmk
    cmk -->|"wraps"| dek
    dek -->|"encrypts"| data

Without an HSM the top edge changes and nothing below it does: the seal cluster is sealed by operator-held Shamir shares, or by a TPM-backed key on each seal node (the mechanism Keystone’s PKCS#11/TPM KEK plan already describes). This is the concrete meaning of “the HSM is optional”.

6.2 Key states

A key moves through a state machine modelled on NIST SP 800-57 and on KMIP’s object lifecycle. The names below are the KMS’s own; the KMIP surface maps them onto the states its clients expect (enabled to Active, disabled and pending_deletion to Deactivated, deleted to Destroyed, compromised to Compromised), and that mapping is part of the KMIP conformance suite rather than an assumption:

¹ Only under an explicit recovery grant, and every use is a high-severity audit event.

Key versions are independent of key state: rotation adds a version, the previous versions stay decrypt-capable until the tenant explicitly retires them, and min_decryption_version (the same concept OpenBao’s transit engine uses) lets a tenant close off old ciphertext deliberately.

The waiting period on deletion is non-negotiable and non-shortenable by any API. Immediate destruction is a kms-manage operation with quorum approval, because “delete the key” is the one KMS operation that destroys customer data irreversibly and is therefore the prime target of a compromised tenant account.

6.3 How a CMK is stored, and how it is used

OpenBao’s transit engine deliberately does not let a key leave the barrier — that is its whole value, and it is also why the offload of §7 cannot be built on transit alone. The design therefore stores a CMK twice over, and the choice of which representation is authoritative is per key:

A key created in mode A has only the first representation and can never be promoted to mode B without a rotation, which is what makes “this key never leaves the barrier” a statement the KMS can keep. A key in mode B has both: the transit key remains the authority for rotation and for the audit trail, and the wrapped blob is the artifact the crypto tier consumes. Mode D replaces both with an HSM object handle (§10).

Which of the two is authoritative has to be answered explicitly, because both obvious answers are wrong. The transit key is not the authority: it cannot produce the blob, since transit does not export, which is the premise this whole subsection rests on. The blob is not the authority either: it knows nothing of versions the tenant has retired. The authority is the key’s metadata row in the control plane (§16). The row records which versions exist and which representations each version is supposed to have, and a version does not become enabled until the row says every representation it needs has landed. Creating or rotating a mode-B key is therefore one operation staged through that row, not two independent writes to the shard.

Divergence is then repaired toward the row, and in one direction only:

• A representation the row does not know about is the debris of a failed rotation, and is destroyed.
• A representation the row expects but cannot find puts the version in unavailable (§6.2). That is an incident, not a self-healing condition: a missing blob cannot be regenerated from transit, so the only repair is a restore from the shard’s backup.

The custody snapshot of §15.2 already walks this inventory every hour, which makes it the right place to detect a discrepancy — rather than discovering it on the next cache fill, in front of a tenant.

This is the mechanism the rest of the document assumes; it is spelled out here because “cache the KEK” is otherwise an instruction that cannot be carried out against transit.

6.4 Envelope encryption

The contract, identical in the native API, the Vault surface, and KMIP:

sequenceDiagram
    participant C as Client
    participant A as kms-api
    participant X as kms-crypto
    participant S as Shard (OpenBao)

    C->>A: POST /v1/keys/K/data-key {spec: AES_256, aad}
    A->>A: authn (SVID/JWT), authz (Keystone check, cached), rate limit
    A->>X: generate_data_key(key=K, ctx)
    alt KEK cached and warm
        X->>X: derive DEK from the cached KEK
    else cold
        X->>S: kv read of the wrapped CMK blob for K@v
        S-->>X: wrapped CMK
        X->>S: transit/decrypt with the shard wrapping key
        S-->>X: plaintext KEK
        X->>X: admit to cache (TTL, use count), derive DEK
    end
    X-->>A: {plaintext_dek, wrapped_dek, key_version}
    A-->>C: 200 {plaintext, ciphertext_blob}
    Note over C: Client encrypts data with plaintext DEK,<br/>discards it, stores ciphertext_blob alongside data

Three properties that matter:

• The AAD is mandatory and structured. Every wrap binds (project_id, key_id, key_version, purpose, caller_scope_digest) as additional authenticated data. A wrapped DEK for project A’s volume cannot be unwrapped in project B’s context even if the blob leaks, and a DEK minted for volume encryption cannot be replayed into an object-storage context.
• The ciphertext blob is self-describing and versioned: a header with format version, algorithm, key ID, key version, region, and the wrapping mode. Crypto-agility lives here.
• Nothing about the shard appears in the blob. Shard identity is metadata the control plane owns, so that migrating a tenant between shards does not invalidate a single stored ciphertext.

6.5 Algorithms

The KMS never invents cryptography. Symmetric and AEAD primitives come from vetted crates (aws-lc-rs or ring, with a FIPS-validated module where the deployment needs it); asymmetric operations are either delegated to OpenBao’s transit engine or to the HSM. Which of those three performs a given operation is a deployment decision recorded per key, not a code path the tenant can select.

7. Offloading envelope encryption from the OpenBao fleet

This is the part of the design that decides whether the service scales.

7.1 The problem

A naive implementation forwards every encrypt/decrypt/data-key call to the tenant’s shard. Then:

• Every object write in an encrypted Swift container, every Cinder attach, every TDE page-key fetch, every customer application call is an OpenBao request. OpenBao’s transit engine is fast, but it is also the thing that must stay up for key management, and it is backed by Raft — read-heavy load and leader elections do not mix well under stress.
• The shard becomes both the availability bottleneck and the latency floor for the entire region’s data path.
• Scaling means more shards, which is the expensive axis.

7.2 The options

Recommendation: B as the default, C for high-volume service integrations, D where an HSM is present and the tenant pays for it, A always available as a per-key opt-out. The choice is a property of the key (offload_mode), visible in the key’s metadata, chosen at creation, and changeable only through a rotation. A tenant with a compliance requirement that key material never leave the barrier selects A and accepts the latency.

7.3 How the cache is made safe

The cache holds plaintext key-encryption keys — calling it anything softer would be dishonest, and it is the single most security-sensitive component in the design. Its rules are therefore explicit:

1. Bounded lifetime. A cache entry has a hard TTL (default 300 s) and a hard use count. Both are per-key configurable downward, never upward.
2. Memory hygiene. Plaintext key material lives in a zeroizing, mlock-ed allocation. The process disables core dumps and ptrace attachment, and runs with swap disabled or encrypted.
3. No persistence. The cache is never written to disk, never included in a heap profile, and does not survive a restart. A kms-crypto restart is a cache flush and a self-healing action.
4. Revocation is push, not poll. Disabling a key, revoking a grant, scheduling deletion, or a tenant’s “panic” call publishes an invalidation to every kms-crypto replica over the internal event bus, and the operation does not return success until every live replica has acknowledged or been fenced. This is what makes “disable my key” mean something within seconds rather than within a TTL.
5. Per-key opt-out is honoured end to end. A key in mode A is never admitted to the cache; the admission check happens in one function, and that function is unit-tested against every mode.
6. Isolation by shard. Cache partitions are keyed by shard, so that a compromised or misbehaving shard’s entries can be evicted wholesale, and so that cache capacity planning follows the shard model.
7. An attestation is never cached by proxy. The cache holds key material, keyed by key and shard, and never an authorization decision. That separation is what lets attestation-bound release (§2.6) cost nothing on the hot path, and it is also what forbids the shortcut: a cached authorization decision may not outlive the attestation result it was based on, and an attestation-bound operation is re-decided when that result expires, whatever the KEK entry’s TTL says.

The degraded-mode consequence is deliberate and is the answer to §1’s second question: if a shard becomes unavailable, kms-crypto keeps serving envelope operations for keys whose entries are warm, for the remainder of their TTL, while key management fails. A storage system stays readable through a short backend incident. This is a considered trade — a longer TTL buys availability and costs revocation latency — and it is an operator-tunable number with a documented meaning, not a hidden default.

7.4 Reducing load further

• Batch everything. One request, N operations, one authorization decision, one audit event with N sub-records.
• Per-request cache (ADR 0030) for authorization decisions and key metadata within a batch.
• Cache ciphertext-to-version mapping so that rewrap storms after a rotation are spread over time rather than stampeding.
• Admission control before the backend, not after. Rate limiting (ADR 0022) is evaluated in kms-api handlers; a rejected request never reaches the crypto tier, let alone the shard.

8. The OpenBao fleet: multitenancy, sharding, placement

8.1 Why a fleet

One cluster per region is simpler and unacceptable: it makes every tenant’s availability depend on every other tenant’s load, makes upgrades a region-wide risk, makes a data-corruption incident unbounded, and caps scalability at one Raft group’s write throughput.

8.2 The shard model

A shard is one OpenBao cluster (3 or 5 nodes, integrated Raft storage, one barrier key, one seal configuration) plus its metadata in the control plane. A shard hosts M tenants, where M is an operational parameter (initially in the low hundreds) chosen so that:

• the shard’s Raft state stays comfortably within memory and snapshot budgets;
• restoring the shard from backup is an operation measured in minutes;
• the customer-visible blast radius of losing it is acceptable to the cloud’s own SLA and communication plan.

Each tenant gets its own OpenBao namespace (or, where namespaces are not used, a strictly separated mount and policy set) inside its shard. Within the namespace: one transit mount for keys, one kv v2 mount for secrets. Policy inside the shard is generated by the control plane; no human writes it.

8.3 Placement

Directory placement is recommended because the interesting requirements are not expressible as a hash: “this tenant is regulated and gets a dedicated shard”, “this tenant is large and must not share with another large one”, “this tenant is in the free tier and goes on the dense shards”, “drain shard C for decommissioning”. The directory is small, changes rarely, and is cached aggressively — but it is on the path of every key operation that misses the crypto tier’s cache, alongside the key’s metadata and the authorization decision, so it is among the first things to be made highly available and the first to be made cheap to read.

Placement classes, as an initial vocabulary: standard (dense, shared), dedicated (one tenant per shard), regulated (dedicated + HSM-backed root + restricted operator access), internal (the cloud’s own services, never shared with tenants).

Placement class also decides the shard type, and that is what makes the custody backend pluggable without inventing a second mechanism. standard, dedicated and internal place a tenant on an OpenBao shard; regulated may place one on a shard whose custody is an HSM partition plus its control-plane metadata (§10.1). The directory already resolves project -> shard, so routing to a different kind of shard is a property of the row rather than a new abstraction, and migration (§8.4) is already the procedure for moving between shards. Shard type constrains what may be created on it — an HSM shard holds key counts in the thousands and supports only hsm-resident keys in modes A or D — and kms-control rejects an impossible request at creation, in the same validation that already enforces the (protection_level, offload_mode) pairing of §10.3.

8.4 Migration between shards

Migration must exist from the start, or the fleet ossifies. The procedure:

1. Control plane marks the tenant migrating, freezes key-management operations (data-plane operations continue).
2. Existing key versions are moved by kms-crypto: for a mode-B key it reads the wrapped blob, unwraps it under the source shard’s wrapping key and re-wraps it under the destination’s, in memory, never writing plaintext anywhere in between. A mode-A key has no exportable representation by construction, so it is migrated by re-creating the transit key at the destination from the same wrapped blob held for this purpose, or — where the tenant chose mode A precisely to forbid that — it is not migrated, and draining its shard requires the tenant’s participation. That asymmetry is the price of mode A and is documented to the tenant at key creation.
3. Verification pass: every key version is read from the destination and compared by digest of a deterministic test vector, not by comparing key material.
4. Directory flip, cache invalidation, source namespace sealed and later destroyed after a retention period.

Because no ciphertext blob names a shard (§6.4), no tenant data needs to be re-encrypted. That property is bought in §6.4 and spent here, and it is the reason for the constraint.

8.5 The fleet as a control loop

kms-fleet is a reconciler, not a script collection. Desired state — shard count, version, seal configuration, tenant assignment — is declarative; the reconciler creates clusters, initializes them, configures seals, applies policy, runs upgrades one node at a time with quorum checks, takes and verifies backups, and reports drift. Every action it takes is an audit event with the same schema as an API call, because “what did the fleet do at 3 a.m.” is an audit question.

8.6 The OpenBao contract, deliberately narrow

The service depends on a stated subset of OpenBao and nothing else. This table is the contract; anything outside it is a feature the KMS may use but must be able to lose:

Everything richer in the transit engine — per-key rotation and versioning, rewrap, datakey, signing, HMAC — is used by mode-A keys only. That is deliberate containment rather than an accident of the default: mode A is the tenant-visible opt-out that trades throughput for “key material never leaves the barrier”, and it is also the one place where the design takes a hard dependency on transit as a cryptographic engine rather than as a barrier. Losing the OpenBao dependency would therefore cost the service mode A and the kv surface of open question 5; it would not cost the service.

Two rules keep the contract from eroding by accretion, which is how such contracts normally die:

1. The dependency lives in one crate. No code outside kms-bao (§17) constructs an OpenBao path, parses an OpenBao response, or knows that namespaces exist. A leak of OpenBao vocabulary into kms-api or kms-control is a review failure, not a shortcut.
2. Nothing OpenBao generates is durable state anywhere else. Shard identity never appears in a ciphertext blob (§6.4), OpenBao token and lease lifetimes are never the basis of an authorization decision (§13), and every mapping that must survive a shard is a row in the control plane’s own database (§16).

9. Sealing, unsealing and the root of trust

9.1 The requirement

Shards must come up unattended — after a node reboot, an AZ failure, a rolling upgrade, or a full region cold start — without a human typing unseal keys, and without depending on any KMS outside this cloud.

9.2 The seal hierarchy

flowchart LR
    subgraph tier0["Tier 0 — root"]
        hsm["HSM partitions<br/>(optional)"]
        shamir["Shamir shares<br/>(operator-held, offline)"]
        tpm["TPM-sealed node keys<br/>(per seal node)"]
    end
    subgraph tier1["Tier 1 — seal cluster"]
        seal["Central OpenBao<br/>transit mount 'fleet-unseal'"]
    end
    subgraph tier2["Tier 2 — fleet"]
        a["Shard A"]
        b["Shard B"]
        c["Shard C"]
    end
    hsm -->|"pkcs11 seal"| seal
    shamir -.->|"or"| seal
    tpm -.->|"or"| seal
    seal -->|"transit seal,<br/>one key per shard"| a
    seal --> b
    seal --> c
    hsm -.->|"direct pkcs11 seal<br/>(alternative)"| a

Tier 1 exists to avoid N HSM client configurations and to give one place where “which shard may unseal” is authorized and audited. Each shard has its own transit key in the seal cluster, and a shard may unseal only against its own key.

The same authentication question as §5.6 arises here and has to be answered differently, because here the seal client is a stock OpenBao node, not something the KMS wrote: a shard node’s seal stanza needs a credential for the seal cluster at boot, before anything else on that node is running. The options, and why the last one wins:

With the third, the material on a shard node’s disk is a SPIRE agent configuration rather than a seal credential, so a stolen backup of a shard’s storage is useless without an identity the seal cluster accepts — but the property is bought by the credential helper, not by the seal stanza, and the cold-start order in §9.4 has to place SPIRE before the shards for exactly this reason.

9.3 Options for the root

Recommendation: TPM-sealed seal-cluster nodes as the no-HSM baseline (it reuses the mechanism Keystone’s PKCS#11/TPM plan already builds), with an HSM seal as a configuration change for deployments that need it. Shamir shares remain as the offline recovery path, held under split custody, exercised in drills. Attested nodes are listed because a deployment that already runs confidential compute has the mechanism to hand and may prefer it to TPM sealing; it competes on operational cost rather than on capability, and principle 6 forbids it becoming the only way to start a region.

9.4 Bootstrap: the chicken and the egg

Cold start of an entire region has a strict order, and it must be written down because it is the procedure nobody practises until they need it:

1. HSM partitions online, or a quorum of seal-cluster nodes with intact TPM state, or operators with Shamir shares.
2. Seal cluster unseals. It depends on no running service — in particular not on Keystone, not on a live SPIRE server, and not on the KMS’s own database. It does hold a static copy of the SPIRE trust bundle so that it can verify the shard credentials of step 4 once they appear; a static trust anchor on disk is not a runtime dependency, and keeping it to that is a hard constraint on the seal cluster’s configuration.
3. SPIRE server comes up. Its own CA key may be KMS-held only if the KMS can serve it without SPIRE — which it cannot in the general case, so in Phase 1 SPIRE keeps its own root and the dependency is left uninverted (§9.6).
4. Shards unseal against the seal cluster using node identities.
5. kms-control connects to its database and to the shards.
6. kms-crypto starts with a cold cache; kms-api starts last and only reports ready when a canary key round-trips end to end.

The readiness probe of kms-api performing a real cryptographic round trip against a canary key in each reachable shard — rather than returning 200 because the process is alive — is what keeps a half-initialized region out of the load balancer.

9.5 Seal rotation and rekey

Rotating a shard’s transit unseal key, rotating the seal cluster’s root, rekeying a shard’s barrier (rekey), and rotating the HSM’s root are four distinct operations with four distinct runbooks, all driven by kms-fleet, all quorum-approved, all rehearsed on a schedule. A KMS that cannot rotate its root of trust has no answer to a compromise.

9.6 Reflexive dependencies

The KMS must be careful about becoming the keeper of the keys that it itself needs. Three cases:

• SPIRE’s CA key. Tempting, and circular: SPIRE issues the SVIDs that the KMS uses to authenticate its own tiers. Phase 1 keeps SPIRE’s root separate (HSM or TPM). A later phase may invert this if the KMS’s seal path is proven to work with certificate-less bootstrap for its own tiers.
• Keystone’s signing keys (ADR 0026). Also circular, since the KMS authorizes against Keystone. Same resolution: keep separate initially; revisit once the KMS has a “break-glass, no external authorization” internal mode that is itself safe.
• The KMS’s own database encryption. Solved without recursion: the metadata database holds no key material (§16), so it needs only transport and at-rest disk encryption, whose key comes from the same TPM/HSM tier as the seal cluster.

10. HSM integration

HSM support is a capability of a deployment, expressed per key and per shard, and it is never required for the service to function.

10.1 Three integration points

10.2 Mechanism

PKCS#11 via a vetted Rust binding, loaded as a dynamic module configured per deployment; no HSM vendor SDK is linked into the binary. The same abstraction carries a TPM (tpm2-pkcs11) and a software HSM (SoftHSM) so that CI and development exercise the same code path as production. This is exactly the approach Keystone’s PKCS#11/TPM KEK plan takes, and the crate is shared.

Sessions are owned by kms-crypto (never by kms-api), pooled, health-checked and re-established on partition failover. HSM throughput is finite and much lower than software crypto, so an HSM-backed key is rate-limited on its own budget and the limit is a property of the key, visible to the tenant, rather than an opaque cause of tail latency.

10.3 The honest documentation requirement

Whatever the deployment does, the key’s metadata states its protection level and the API returns it:

protection_level and offload_mode (§7.2) are not independent: an hsm-resident key has no representation outside the HSM, so it is restricted to mode A (every operation in the HSM) or mode D (DEK generation in the HSM), and the API rejects any other combination at creation rather than silently weakening one of the two. hsm-sealed places no constraint on the offload mode, because the seal chain is about how the barrier is protected, not about where the key is used.

A customer choosing a key’s protection level and being able to verify it later is a requirement, not a feature. So is refusing to let protection_level be downgraded by any API.

11. Regions, high availability and disaster recovery

11.1 Regions are hard boundaries

A key belongs to a region. Its identifier carries the region. Its material never leaves the region unless it is a multi-region key and the tenant requested replication. There is no global control plane, no global database, and no cross-region synchronous call on any data path. This mirrors the region model OpenStack already has and avoids the failure mode where a WAN partition takes out key operations in a healthy region.

11.2 Multi-region keys

Replication moves key material between regions, so it happens between kms-crypto tiers over mutually authenticated SPIFFE mTLS with a federated trust domain, wrapped under a region-pair wrapping key, with both regions recording the transfer. A tenant must opt in per key, and the key’s metadata permanently records every region it has ever been replicated to — you cannot un-replicate material, and pretending otherwise would be dishonest.

11.3 In-region HA

• kms-api, kms-crypto: stateless, N+2 across AZs, drained on deploy.
• kms-control: the metadata database is synchronously replicated across AZs; schedulers are leader-elected.
• Shards: 3 or 5 nodes across AZs; loss of one AZ leaves quorum. An AZ that cannot reach the seal cluster cannot unseal, so the seal cluster is also AZ-distributed and is the first thing recovered.
• Load balancing: kms-crypto prefers the replica whose cache is warm for the requested key (consistent hashing on key ID with bounded load), falling back to any replica. Cache warmth is a performance property; correctness never depends on hitting a particular replica.

11.4 The Keystone dependency

Authorization is a live decision. That is right for a KMS — a revoked grant must stop working — but a Keystone brown-out must not become a data outage. The posture:

• Decisions are cached per (subject, key, operation) with a short TTL (default 60 s for management operations, configurable up to a few minutes for data operations).
• The check API is served from a region-local Keystone read path (Raft learners or read replicas, per vision §8).
• Fail closed by default. When no cached decision applies and Keystone is unreachable, the request is denied.
• A narrowly scoped, explicitly enabled, time-boxed and loudly audited “extended cache” mode lets an operator raise the data-path TTL during a Keystone incident. It is an incident action with an expiry, not a configuration default, and while it is active every served request carries a header saying so and every audit event records it.

11.5 Backup and disaster recovery

• Shard snapshots are taken by kms-fleet, encrypted under a backup key held in the seal cluster (and in the HSM where present), stored in at least two failure domains, and restore-tested on a schedule into an isolated environment. A backup that has never been restored is not a backup.
• The control-plane database is backed up with the same discipline; it holds no key material, so its exposure is metadata exposure — still serious, still encrypted.
• The recovery objectives are stated per component and published to tenants, because the recovery time of a KMS is the recovery time of everything encrypted with it.
• A documented, rehearsed region cold-start drill (§9.4) at least annually.

12. Network isolation

The zones of §5.1 map onto networks. This is where the “public versus internal” requirement becomes enforceable rather than aspirational.

flowchart TB
    subgraph n1["Public network (internet / tenant)"]
        lb1["Public LB :443, :5696"]
    end
    subgraph n2["OpenStack internal network"]
        lb2["Internal LB :8443"]
        svcs["Cinder, Glance, Nova, Swift"]
    end
    subgraph n3["KMS service network"]
        api["kms-api"]
        cry["kms-crypto"]
        ctl["kms-control"]
    end
    subgraph n4["Backend network"]
        shards["OpenBao shards"]
        db[("Metadata DB")]
    end
    subgraph n5["Seal network"]
        seal["Seal cluster"]
        hsm["HSM"]
    end
    subgraph n6["Management network"]
        fleet["kms-fleet, kms-manage"]
    end

    lb1 --> api
    lb2 --> api
    svcs --> lb2
    api --> cry
    api --> ctl
    cry --> shards
    ctl --> shards
    ctl --> db
    shards --> seal
    seal --> hsm
    fleet --> ctl
    fleet --> shards
    fleet --> seal

12.1 Rules

1. No tenant-routable path to the backend network. The shards have no route to, and no listener reachable from, any tenant network. This is enforced by routing and by firewall/NetworkPolicy, and verified by an automated reachability test that runs in CI against the deployment, not by a diagram.
2. The seal network is the most restricted. It carries exactly two traffic patterns: shard nodes to the seal cluster, and the seal cluster to its HSM partition. kms-api has no route to it. Neither does kms-control, except through kms-fleet on the management network. The HSM used for crypto offload (§7 option D, §10.1) is a different partition reached over a different path from the KMS service network, and the two must not be collapsed into one client configuration just because they are the same appliance: the seal partition is reachable only by the seal cluster, and a compromise of kms-crypto must not yield a seal credential. Where only one HSM exists, this is enforced by separate partitions with separate credentials, and it is a deployment review item.
3. Each hop is mutually authenticated. SPIFFE mTLS everywhere inside; network controls are defence in depth, not the primary control. A compromise of a network boundary should not by itself yield key material.
4. Egress is default-deny, and the allowed list is exhaustive. kms-crypto may reach the shards, its HSM crypto partition, the invalidation bus and the audit pipeline. kms-api may reach kms-crypto, kms-control, Keystone, OPA and the audit pipeline. kms-control may reach the shards, its database, the invalidation bus and the audit pipeline. Nothing else, in either direction, for any of them. A KMS with unrestricted egress is a KMS with an exfiltration path.
5. Separate listeners, separate certificates, separate CAs. The public listener uses a public CA; internal listeners use the SPIFFE trust domain. No certificate is valid in two zones.
6. KMIP gets its own address and its own budget. KMIP clients are long-lived connections with different failure behaviour from HTTP; sharing a process is fine, sharing a connection budget is not.
7. Administrative interfaces are never on the public LB. No exceptions, no “temporarily”, no IP allowlist as a substitute.

12.2 Multi-tenant traffic hygiene

• Per-project connection and request budgets at the edge, so one tenant cannot exhaust the shared front end.
• Separate rate-limit governors per zone (ADR 0022): a tenant burst and a Cinder mass-attach draw on different budgets.
• Request and response size caps on every endpoint, enforced before parsing.
• The KMIP front end drops connections that do not complete a TLS handshake with an acceptable client certificate within a short deadline; unauthenticated TTLV is never parsed.

13. Authentication and authorization

13.1 External callers

Exactly as in vision §4.3 and §5: a scope-less JWT (or an SVID at the TLS layer) says who, the addressed key says where, and the operation says what. Ownership is never read out of the credential: the key’s owning project comes from the KMS’s own metadata, which is the authoritative statement, and a credential that names a project does not thereby gain access to it.

KMIP is the one surface where that has to be qualified, and the qualification matters enough to state rather than bury. KMIP has no field in which a client can name a target, so the connection is bound to a project derived from the client identity at handshake time (§5.5). That derived project is a default target, not an authorization: the KMS still evaluates the same per-object policy it would for a REST call, and an object the principal may not read is invisible to Locate and denied to Get. The binding chooses which target the request is about; it never decides whether the request is allowed.

13.2 Internal callers

SPIFFE mTLS peer identity, plus — when acting for a user — the user’s JWT or an on-behalf-of exchange. The KMS refuses a service-only identity for any operation on a tenant key unless the tenant has granted that service a standing grant (for example “Cinder may unwrap DEKs for key K”), because a service identity that can unwrap any tenant key is precisely the design flaw this architecture exists to avoid.

13.3 Policy

OPA, with the bundle layout the repository already uses: policy/kms/<resource>/<action>.rego — policy/kms/key/create.rego, policy/kms/key/decrypt.rego, policy/kms/grant/create.rego, and so on. The policy input carries the authentication chain, the operation, the key’s metadata (owner, state, protection level, offload mode), the grant set, and the request’s context — and never key material, wrapped or otherwise. That exclusion is a test, not a convention.

Rules that follow directly from the security model:

• Decisions key on the authentication chain, never on a caller-supplied scope.
• Delegated operations compare against input.credentials.delegated_project_id.
• The scope-drift tripwire applies unchanged.
• Every list endpoint re-checks each key individually with the per-item read policy.
• Sensitive operations (deletion scheduling, export, grant creation, protection downgrade attempts) may require amr containing a phishing-resistant method, expressed as a policy rule rather than a separate credential type.

13.4 Grants

A grant is the KMS’s unit of delegation: “principal P may perform operations O on key K, optionally constrained by encryption context, optionally expiring”. It is deliberately the same shape as the permission grants of vision §6, and where Keystone gains resource-level targets (vision §6.5) the KMS is the first consumer: a grant on a key is a resource-level grant, and modelling it in Keystone rather than in the KMS’s own database is the preferred end state.

The condition slot is where attestation-bound release lands (§2.6): principal P may unwrap key K if it presents an attestation result satisfying appraisal policy A. The KMS verifies the result’s signature against a verifier it has been configured to trust and evaluates its claims in Rego; it does not appraise raw evidence. Attaching the condition to the grant rather than to the key is the deliberate part. A caller that cannot produce a result finds the grant unusable, which is the intended failure mode, while the key itself stays usable by the tenant’s ordinary tooling — so a tenant who mis-specifies an appraisal policy has a broken workload, not a key it can no longer reach.

Two implementation options:

Recommendation: store grants in the KMS initially with a schema that maps one-to-one onto the Keystone model, and migrate when vision §6.5 lands. The migration is a data move, not a redesign, precisely because the schema was chosen for it.

13.5 Quotas

Per project: number of keys, key versions, grants, secrets, operations per second by operation class, HSM-backed key count. Quota enforcement happens at the edge, is visible through /v1/quotas, and is a first-class defence against a compromised tenant account driving cost or backend load.

14. Audit, metrics and transparency

• CADF events (ADR 0023) naming the initiator, the acting service if any, the key, the key version, the operation, the outcome, the encryption context digest (never the context’s secret parts), and the protection level. Three tiers, per principle 10: (a) management operations, every failure, every denial and every operation that touches key material — one event each, never aggregated, never sampled; (b) successful data-plane operations — recorded as per-(principal, key, operation, minute) aggregates carrying a count, so that “who used this key and when” stays answerable at a resolution of a minute without one event per object write; (c) a per-batch event for a batched request, with the item count and any per-item failures enumerated. An operator can promote a key or a project to tier (a) for everything, which is what an investigation needs and what a regulated tenant may pay for.
• Tenant-visible audit. A tenant can read the audit events for their own keys. This is not a convenience feature: “who decrypted my data, and when” is the question a KMS exists to answer, and a KMS that only answers it to the operator is not usable for a regulated customer.
• Metrics (ADR 0031): operation rate and latency by class and key, cache hit ratio and age distribution, backend call rate per shard, shard health and seal status, HSM session health and saturation, authorization decision latency and cache hit ratio, rate-limit rejections. Per-tenant cardinality is bounded deliberately; a tenant label on a high-cardinality metric is a monitoring outage waiting to happen.
• No key material in any telemetry, ever. Enforced by a redaction layer with its own tests, and by the same discipline the audit pipeline already applies to PII.

15. Metering and billing

15.1 Why this belongs in the architecture

Metering is in this document, and not deferred to a commercial annex, for one reason: two of its requirements cannot be retrofitted. The billable party (§15.4) has to be derivable from the authentication chain, which means it is decided by the same model as authorization; and custody metering (§15.2) needs an inventory of key versions that only the control plane can produce. Everything else about pricing can change quarterly. These two cannot change at all once keys exist and invoices have been sent.

The second reason is a correction: audit is not a billing source. §14 deliberately aggregates successful data-plane events per minute and samples nothing else; that is right for answering “who used this key”, and wrong for an invoice, which needs completeness, deduplication and a retention period set by the tax authority rather than by the security team. The two streams share a vocabulary and nothing else.

15.2 What is metered

Three meters, because the service has three genuinely different cost drivers:

Custody is metered per key version, not per key, because a version is what occupies a barrier and a backup. Rotation therefore has a visible price, which is the honest signal: a tenant rotating hourly is buying something real.

The three meters are emitted by three different components on purpose. A discrepancy between the operation stream and the custody snapshot is the cheapest available detector for a metering pipeline that has silently stopped, and reconciliation between them should be a scheduled job, not an incident response.

15.3 Where metering happens, and what it may never do

Operation metering happens at kms-api. That is the only point where the native, Vault-compatible, Barbican-compatible and KMIP surfaces converge, and — more importantly — it is the only point that sees operations served from the kms-crypto KEK cache. An operation satisfied entirely from cache performs no backend work and must still be billed; metering at the crypto tier or at the shard would make the §7 offload look like a discount. The KMIP handler is the one that needs care: a KMIP session is long-lived and multiplexes many operations, so records are emitted per operation, never per connection.

Two rules constrain the implementation:

1. A metering failure never fails a cryptographic operation. A KMS that refuses to decrypt because the rating pipeline is unavailable has converted a billing outage into a data outage. Records are written to a durable local queue and drained asynchronously; when the queue fills, the oldest records are spilled to disk and, past that, revenue is lost in preference to availability. This is a deliberate trade and it is the reason for the reconciliation job above.
2. At-least-once, with an idempotency key. Every record carries (region, emitter_id, monotonic_sequence); the rating system deduplicates. Retried requests, batch items and re-driven queues must not double-bill, and exactly-once delivery is not available to us.

A batch request emits one record per item, with the batch identifier retained — consistent with the per-batch audit event of §14, which already carries the item count.

15.4 The billable party

The authenticated principal is not always who pays. The rule, keyed on the authentication chain exactly as authorization is:

The third row is a product decision with an architectural consequence, so it is stated here rather than left to a price list. When Cinder unwraps a volume key on every attach, that traffic is the cloud’s, generated by the cloud’s own scheduler at a rate the tenant does not control. Billing it as tenant KMS operations charges a customer for an implementation detail and makes encrypted volumes look arbitrarily expensive; the cost belongs in the volume price. What is billed to the tenant is the custody of the customer-managed key itself.

Grants default to the key owner paying because the alternative — a project being billed for a key it does not control and cannot delete — produces disputes that no amount of usage data resolves. The billing = grantee option exists because the opposite arrangement is what a service-provider tenant actually wants.

15.5 Billing semantics of key state

The state machine of §6.2 needs a billing column, and the answers are not self-evident:

Custody is metered hourly and prorated. A key created and destroyed within an hour is billed for an hour — a minimum charge that removes any incentive to churn keys in order to defeat the meter.

15.6 A reasonable tariff

The architecture argues for a specific weighting, and the argument is worth recording even though the numbers are not the architecture’s to set. §7 exists to make a data-plane operation cost approximately nothing at the margin: a cache hit performs no backend call and no shard I/O. Meanwhile the cost the fleet actually incurs is the one computed in §4.3 — quorum × availability zones × shard — which a tenant incurs by existing on a shard, not by issuing requests. A tariff weighted toward per-operation charging would therefore tax the one resource the design deliberately made free, and would reward a customer for defeating the cache the security model already constrains (§7.3).

So: custody dominates, operations are priced as a fairness and abuse signal, dedicated capacity is a flat charge. Indicative anchors, benchmarked against the public clouds this service is a substitute for, and not a price list:

The free allowance is not marketing; it is the same decision as the free-tier placement class of §8.3, and the two should be defined once.

The Phase 1 economics do not close on this tariff, and that is worth stating plainly. An auto-unseal customer needs exactly one key and performs on the order of ten operations a month. A thousand such customers, priced per key and per operation, do not pay for one region’s quorum, let alone the engineering. Phase 1 is therefore priced either as a flat per-project subscription for the capability, or — more honestly — is not a profit centre at all, but the capability that makes customer-managed encryption for Cinder, Swift and Manila possible and regulated workloads onboardable. Which of the two it is should be decided before Phase 1 ships, because it determines whether usage-based billing is needed in Phase 2 or can wait for Phase 3.

15.7 Regions, rollup and replicated keys

§4.3 forbids a global write path, so metering is regional and there is no transactional global view of a tenant’s usage. The consequences:

• Each region emits to its own durable queue and is drained into a rating system outside the KMS’s trust boundary; late and out-of-order arrival is normal and the rating system, not the KMS, closes the billing period.
• A replicated key (§11.2) exists in more than one region and is metered in each, because it occupies a barrier in each. The rating system must deduplicate on the logical key identifier for display while summing the per-region custody records for charge — a tenant reading their bill should see one key charged twice with the regions named, not two keys.
• A region that is isolated continues to serve and continues to meter. Its records arrive when the partition heals. Nothing about billing may become a reason to fail closed.

15.8 Quotas are not entitlements

§13.5’s quotas exist to stop a compromised tenant account from driving cost and backend load. They are a defence, and they are enforced at the edge where that defence is cheap. A billing plan is a different thing, and the two are linked by exactly one rule: a plan may raise a quota, and a quota may never be the enforcement point for a payment decision. Suspension for non-payment is a lifecycle action on the project, taken by the cloud’s billing system through the management zone, audited as a management operation, and — critically — never expressed as a silent quota of zero, which would be indistinguishable to the tenant from an outage of the service they are disputing the bill for.

15.9 What the tenant can see

A GET /v1/usage endpoint on the public zone, returning the current billing period’s custody and operation counts per key and per operation class, from the same records that produce the invoice. The tenant-visible audit of §14 answers “who used my key”; this answers “why is my bill this number”, and a customer who cannot answer the second question without a support ticket will not adopt a KMS for anything that matters.

16. What lives where: storage model

The invariant worth restating: the control-plane database contains no key material. A full dump of it discloses what keys exist, who owns them and how they are used — serious, and a reason to encrypt and restrict it — but it does not disclose a single byte that decrypts customer data.

17. Implementation shape

A Rust workspace mirroring keystone-rs conventions, so that a contributor moving between the two projects finds the same structure. The five processes of §4.2 are five binaries in it; the rest are libraries:

The backend seam. Custody and cryptography are two traits, not one backend. KeyCustody answers “where does this key’s material live, and who may destroy it”; CryptoEngine answers “who performs this operation”. They are kept apart because their implementations differ in both cardinality and failure mode — a shard holds millions of key objects at software speed, an HSM partition thousands at hardware speed — and one trait spanning both would collapse either to their intersection, losing the transit versioning and min_decryption_version that §6.2 depends on, or to a union with runtime capability errors, which is worse than two honest implementations:

Capability negotiation is not done with runtime flags. The permitted combinations are exactly the (protection_level, offload_mode) matrix of §10.3 crossed with the shard type of §8.3, validated by kms-control when the key is created. An unsupported combination is a rejection at creation time, never an Unsupported error surfacing on the data path with a DEK request already in flight.

Engineering rules carried over unchanged: no unwrap/expect/println!/ unsafe; thiserror error types; async on tokio; Apache-2.0 headers; CRUD naming on backend traits; at least three tests per CRUD handler (valid auth plus positive and negative policy, invalid auth). Two additions specific to this project:

• Every cryptographic format has a test vector file committed to the repository, and a test that decrypts a blob produced by an older format version. Wire-format compatibility for ciphertext is forever.
• The crypto tier is fuzzed and the KMIP codec is fuzzed, in CI, with a corpus that grows from every parsing bug found.
• The custody seam is proven by a second implementation, in CI. The KeyCustody contract suite runs against kms-bao and against kms-pkcs11 on SoftHSM (§10.2), from the phase that introduces the trait onward. A seam with one implementation behind it is a guess, and it is always discovered to be the wrong guess at the moment a second backend becomes urgent.

18. Failure modes

The row that deserves attention is “shard loses quorum”: the difference between a short storage incident and a customer-visible data outage is exactly the crypto tier’s cache, which is why §7.3 treats the TTL as an availability control as well as a security control.

19. Phased delivery

The guiding rule: each phase ends with something an operator can deploy and a customer can use. No phase is a refactor of the previous one.

Phase 1 — Auto-unseal for customer OpenBao (the first useful thing)

Goal: a customer’s OpenBao auto-unseals against the cloud KMS, using OpenBao’s own native transit seal, with no custom client software.

• kms-api with the public zone only, and only transit/encrypt, transit/decrypt, auth/cert/login, auth/jwt/login, sys/health, sys/seal-status.
• kms-crypto and kms-control as separate processes from the start, even though Phase 1 has no cache for them to hold and no fleet for them to place onto. Collapsing them into kms-api for one phase would put the network split of §12 — the thing that makes the isolation claims true — on the wrong side of a later refactor, and the isolation is the part a first customer is actually buying.
• SVID (mTLS) and Keystone-JWT authentication; OPA policy; per-project rate limits; CADF audit; Prometheus metrics.
• A single shard per region is acceptable here, but built behind the placement directory from day one, so that Phase 3 is a configuration change rather than a migration of the data model.
• Seal cluster and shard unseal (§9), with the TPM-sealed baseline.
• Native POST /v1/keys (create, list, describe, rotate, disable) so that the transit key has a lifecycle in the KMS’s own model rather than only in OpenBao’s.
• kms-manage bootstrap, region cold-start runbook, restore-tested backups.

The choice of use case is deliberate: it is high-value (customers running OpenBao today either use a public cloud KMS, which is exactly what this cloud cannot offer them, or hold unseal keys by hand), it exercises the whole identity and audit stack, and the client side is already written by someone else.

Phase 2 — Envelope encryption and the crypto tier

• The KEK cache and the invalidation bus (§7.3) in the kms-crypto process that Phase 1 already deployed.
• The wrapped-CMK-blob representation of §6.3, alongside the transit key.
• data-key, unwrap, rewrap, batch variants.
• Offload modes A and B per key; mode A default for any key created before this phase.
• Key state machine complete: scheduled deletion with waiting period, versions, min_decryption_version.
• Grants, with the billing attribute of §15.4 decided at grant creation.
• The KeyCustody/CryptoEngine split of §17, with the contract suite running against both kms-bao and SoftHSM in CI. The seam is introduced here, with the arrival of the crypto tier as its second consumer, rather than retrofitted later once three API surfaces have grown into the first implementation.
• Metering (§15): operation records emitted at kms-api — including cache hits — with idempotency keys and a durable local queue that never fails an operation; the hourly custody snapshot in kms-control; the reconciliation job between the two. The billable-party rule and the per-state billing semantics land here because they are properties of the grant and key-state models this phase completes, not of the rating system. Dedicated shard-hours follow in Phase 3 with the fleet that emits them, and GET /v1/usage can follow whenever the rating system is ready to answer it.

Phase 3 — The fleet

• Multiple shards, placement classes, the directory as the authority.
• kms-fleet reconciler: provisioning, rolling upgrade, backup, health.
• Shard migration with the re-wrap procedure (§8.4).
• Per-tenant namespace isolation and generated policy.

Phase 4 — OpenStack integration

• Internal zone, SPIFFE-authenticated, with delegated user identity.
• Barbican v1 compatibility surface and castellan conformance.
• Cinder, Glance, Swift and Manila integration with customer-managed keys.
• Key-usage attestation and the re-wrap event stream.

Phase 5 — KMIP

• TTLV codec, Baseline + Symmetric Key Lifecycle + Basic Cryptographic profiles.
• Tenant CA registration and certificate-to-principal mapping.
• Interoperability testing against the clients that matter to the cloud’s customers, published as a support matrix.

Phase 6 — HSM and assurance

• PKCS#11 seal, hsm-sealed and hsm-resident protection levels.
• Offload mode D, per-key HSM budgets.
• regulated placement class.
• FIPS-validated cryptographic module option; conformance evidence collection.

Phase 7 — Multi-region and the long tail

• Replicated and primary/replica keys, cross-region replication path.
• BYOK import and authorized export.
• Asymmetric key types and signing at scale.
• Tenant-visible audit API.
• Post-quantum wrapping and signing as the fleet and the ecosystem allow.
• Attestation-bound release for confidential workloads: the grant condition of §13.4, kms-attest, and — only if clients require it — a key-broker surface that speaks the attestation handshake (§2.6).

What could be dropped or reordered

Phases 4 and 5 are independent and can be swapped according to whether the cloud’s near-term demand is OpenStack-internal encryption or enterprise customers with existing KMIP estates. Phase 6 can start earlier if an HSM is already deployed for another purpose; nothing in Phases 1–5 has to change to accommodate it, which is the point of §10.

20. Compliance considerations

Not a compliance plan, but the architectural hooks that a compliance plan will need, so that they are not retrofitted:

• Separation of duties: key custody (fleet operators) is separate from key use (tenants) and from policy administration. kms-manage operations that touch key material require quorum approval.
• Dual control and split knowledge for the Shamir recovery path.
• Key lifecycle states and cryptoperiods aligned with NIST SP 800-57.
• Complete, tamper-evident audit — the HMAC chain the audit ADR already specifies — with tenant-visible export.
• FIPS 140-3 boundary options: a validated software module, or the HSM.
• PCI-DSS and eIDAS-adjacent requirements are deployment questions, but they all reduce to: where is the boundary, who can cross it, and can you prove what crossed it. §12, §13 and §14 are the answers.

21. Open questions

1. Grant authority. Do key grants live in Keystone as resource-level grants (vision §6.5) from the start, accepting the dependency, or in the KMS with a migration later (§13.4)?
2. Cache TTL as a product decision. The KEK cache TTL trades revocation latency against availability during a shard incident. Is the default an operator choice, a per-key tenant choice, or a per-tier product choice?
3. Tenancy inside a shard. OpenBao namespaces versus mount-and-policy separation: namespaces are cleaner but tie the design to a namespace implementation; mounts are universal but produce very large policy sets.
4. Shard sizing. What is M? The answer determines the fleet’s cost and the blast radius simultaneously, and it should be derived from a restore-time measurement rather than from intuition.
5. Vault surface scope. Is kv v2 worth carrying at all, or does the KMS restrict itself to transit and push secret storage to the customer’s own OpenBao — which is, after all, what this service exists to make safe? §8.6 keeps kv inside the depended-on contract but not load-bearing, so dropping it stays a product decision rather than an architectural one.
6. Barbican’s future. Is the compatibility surface a migration bridge with an end date, or a permanent OpenStack-facing contract?
7. KMIP identity for non-SPIFFE clients. Tenant-registered CAs put a certificate-issuance trust decision in tenant hands. Is that acceptable, or should the cloud issue those certificates itself from a per-tenant intermediate?
8. Reflexive dependencies (§9.6). Under what conditions, if any, should SPIRE’s and Keystone’s own keys move into the KMS?
9. Cross-region replication and data residency. Replication is a jurisdictional event, not only a technical one. How is that expressed in policy rather than in documentation?
10. The tariff itself. §15.6 argues for custody-weighted pricing from the cost structure and proposes anchors, but the numbers, the free allowance and whether Phase 1 is a profit centre at all are commercial decisions this document can only frame. The shape influences the API (batching, data-key reuse) more than it looks.
11. A second custody backend. §2.5 rejects both the HSM-only and the no-barrier options for the general case, and §8.6 keeps the OpenBao dependency narrow enough that the decision stays reversible. What would justify actually shipping a second one? Two triggers are worth naming in advance, so that the answer is not improvised during an incident: the fleet’s operational cost coming to dominate the service’s cost structure (§15.6), and a regulated customer whose custody requirement the barrier cannot satisfy at any price. Neither is a Phase 1–3 concern; both are far cheaper to answer if the seam of §17 is real rather than notional.
12. Attestation-bound release. §2.6 recommends owning the release decision and not the appraisal, which keeps the tenant-facing broker a Phase 7 feature rather than an architectural commitment. Two things would move it forward: a confidential-computing offering on the cloud’s own compute plane, which gives the feature a captive first customer, and a customer who would otherwise run their own broker — the same argument §2.5 makes about a customer’s own OpenBao. The service-side use is a separate question with a separate answer: attestation as a Tier-0 root (§9.3) competes with TPM sealing on operational cost, not on capability, and principle 6 already forbids making it a prerequisite.

22. References

• Authentication and authorization vision
• Security model
• ADR 0002 — Open Policy Agent
• ADR 0004 — v4 API
• ADR 0008 — Workload federation
• ADR 0017 — Security context
• ADR 0020 — Mapping engine
• ADR 0022 — Rate limiting
• ADR 0023 — Auditing
• ADR 0026 — OAuth2 / OIDC provider
• ADR 0028 — Quorum-bypass emergency operations
• ADR 0029 — Pagination
• ADR 0030 — Per-request cache
• ADR 0031 — Prometheus metrics
• ADR 0032 — Vendor data JWT attestation
• ADR 0036 — Service delegation
• PKCS#11/TPM KEK plan (doc/plans/0016-v2-pkcs11-tpm-kek.md)
• SPIRE integration plan (doc/plans/spire-integration.md)
• OASIS KMIP 1.4 and 2.1 specifications and profiles
• NIST SP 800-57 (key management), SP 800-38D/F (GCM, key wrapping), FIPS 140-3
• OpenBao documentation: transit secrets engine, seal/unseal, namespaces, integrated storage
• SPIFFE/SPIRE specifications; RFC 8705 (mTLS-bound tokens), RFC 7523

Installation

The easiest way to get started with the keystone-ng is using the container image. It is also possible to use the compiled binary. It can be either compiled locally or downloaded from the project artifacts.

Using pre-compiled binaries

As of the moment of writing there were no releases. Due to that there are no pre-compiled binaries available yet. Every release of the project would include the pre-compiled binaries for a variety of platforms.

Compiling

In order to compile the keystone-ng it is necessary to have the rust compiler available. It may be installed from the system packages or using the rustup.rs

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Afterwards in the root of the project source tree following command may be executed to invoke the cargo

cargo build --release

It produces 2 binaries:

• target/release/keystone (the api server)

• target/release/keystone-db (the database management tool)

Currently keystone depends on the openssl (as a transitive dependency). Depending on the environment it may be a statically linked or dynamically. There are signals that that may be not necessary anymore once all dependencies transition to the use of rustls.

Cargo features

The default build keeps the dependency tree small (for example for distribution packaging). Optional functionality is enabled with --features on the openstack-keystone crate, e.g. cargo build --release -p openstack-keystone --features wasm_plugins.

None of these are enabled by default. CI runs the test suite with --all-features, so every feature is built and tested. A build without wasm_plugins ignores [auth_plugins] configuration and logs a warning at startup.

Using containers

It is possible to run Keystone-ng inside containers. A sample Dockerfile is present in the project source tree to build container image with the Keystone and the keystone-db utility. When no ready image is available it can be build like that:

docker build . -t keystone:rust

Since keystone itself communicates with the database and OpenPolicyAgent those must be provided separately. docker-compose.yaml demonstrates how this can be done.

docker run -v /etc/keystone/:/etc/keystone -p 8080:8080 ghcr.io/openstack-experimental/keystone:main -v /etc/keystone/keystone.conf

Database migrations

Rust Keystone is using different ORM and implements migration that co-exist together with alembic migrations of the python Keystone. It also ONLY manages the database schema additions and does NOT include the original database schema. Therefore it is necessary to apply both migrations.

keystone-db --config /etc/keystone/keystone.conf

It is important to also understand that the DB_URL may differ between python and rust due to the optional presence of the preferred database driver in the url. keystone-ng will ignore the the driver in the application itself, but the migration may require user to manually remove it since it is being processed by the ORM itself and not by the keystone-ng code.

OpenPolicyAgent

keystone-ng relies on the OPA for policy enforcement. Default policies are provided with the project and can be passed directly to the OPA process or compiled into the bundle.

opa run -s policies

NOTE: by default OPA process listens on the localhost only what lead to unavailability to expose it between containers. Please use -a 0.0.0.0:8181 to start listening on all interfaces.

Parallel installation with the python Keystone

Since Keystone-NG is only an addition and is not a drop-in replacement for the python Keystone it is necessary to deploy both versions together.

With the python Keystone no changes on the deployment strategy should be necessary. Whichever WSGI method is used to run the Keystone it stays this way and continues listening on the expected port.

The rust Keystone is deployed in parallel to it (usually on the same hardware) and by default it listens on the port 8080.

Next required step is to alter the http proxy server configuration. In the devstack this is usually the Apache webserver. Some operators may run nginx or haproxy in front of the default webserver with the Keystone. Depending on the preferred functionality (i.e. whether the token validation should be performed by the python or the rust implementation) redirects can be implemented. This way it is possible to decide for every single API call individually whether it should be served by python or rust implementation.

Nginx sample configuration

server {
    listen 443 ssl;
    server_name devstack.v6.rocks;

    ...

    # by default in devstack services are exposed with the url path style.
    location /identity/v4 {
      server http://localhost:8080;
    }
    proxy_pass http://<192.168.1.1>;
}

Apache sample configuration

<VirtualHost *:443>
    ServerName devstack..v6.rocks

    ...
    # Rust /v4 API
    ProxyPass "/identity/v4" http://localhost:8080/v4 retry=0
    # Python /v3 APIs are served by the uwsgi app
    ProxyPass "/identity/v3" "unix:/var/run/uwsgi/keystone-api.socket|uwsgi://uwsgi-uds-keystone-api/v3" retry=0
    # We want discovery URL to be served by Rust. The same way any /v3/ API can
    # be forwarded to rust version (where supported).
    ProxyPass "/identity" http://localhost:8080 retry=0
</VirtualHost>

Devstack integration

The devstack/ directory in the project source tree is a standard out-of-tree devstack plugin that automates the Apache reroute described above. Python Keystone still owns the base DB schema and initial admin bootstrap; the plugin builds/runs the rust binary alongside it, applies the additive DB migrations, and rewrites the Apache site devstack generated for Keystone so that /identity is proxied to the rust binary instead of python’s uwsgi socket.

To use it, add to local.conf:

[[local|localrc]]
enable_plugin key-rs https://github.com/openstack-experimental/keystone
enable_service key-rs

See .github/workflows/devstack.yml for a working example that runs a minimal devstack (mysql, rabbitmq, keystone) in CI with the plugin enabled.

General Information

Keystone-NG is a Rust implementation of OpenStack Identity capabilities. It can run beside Python Keystone, with a reverse proxy routing requests to the implementation that owns each endpoint.

Use this section for information that applies to operators, API consumers, and contributors:

• Architecture describes the service and its dependencies.
• Compatibility explains the relationship with Python Keystone and the v3 and v4 APIs.
• Performance comparison explains how performance results are produced and where to find current measurements.

For task-oriented documentation, continue with the installation guide, user documentation, or administrator guides.

Architecture

Keystone requires 2 additional components to run:

• database (the same as the py-keystone uses)

• OpenPolicyAgent, that implements API policy enforcement

architecture-beta

    service db(database)[Database]
    service keystone(server)[Keystone]
    service opa(server)[OpenPolicyAgent]

    db:L -- R:keystone
    opa:L -- T:keystone

Database

Python keystone uses the sqlalchemy as ORM and the migration tool. It cannot be used from Rust efficiently, therefore keystone-ng uses the sea-orm which provides async support natively and also allows database type abstraction. Current development focuses on the PostgreSQL database. The MySQL should be supported, but is not currently tested against.

New API and resources are being added. This requires database changes. sea-orm also comed with the migration tools. However there is a slight difference between sqlalchemy and sea-orm. The later suggests doing database schema first. In the next step object types are created out of the database. That means that the database migration must be written first and cannot be automatically generated from the code (easily, but there is a way). Current migrations do not create database schema that is managed by the py-keystone. Therefore in order to get a fully populated database schema it is necessary to apply keystone-manage db_sync and keystone-db up independently.

Target of the keystone-ng is to be deployed in pair with the python keystone of “any” version. Due to that it is not possible to assume the state of the database, nor to apply any changes to the schema manaaged by the py-keystone. A federation rework assumes model change. To keep it working with the python-keystone artificial table entries may be created (in the example when a new identity provider is being created automatically sanitized entries are being added for the Python Keystone identity provider and necessary protocols) A federation rework assumes model change. To keep it working with the python-keystone artificial table entries may be created (in the example when a new identity provider is being created automatically sanitized entries are being added for the Python Keystone identity provider together with necessary idp protocols).

Fernet

keystone-ng uses the same mechanism for tokens to provide compatibility. The fernet-keys repository must be provided in the runtime (i.e. by mounting them as a volume into the container). The keystone-manage token setup and keystone-manage token rotate commands manage the repository, and running services reload key changes without a restart. See Fernet tokens for configuration and operational procedures.

Compatibility

Keystone-NG augments Python Keystone rather than requiring an immediate full replacement. Deployments can route selected requests to either service while both implementations use compatible identity data and token formats.

API Compatibility

• The v3 API implements an expanding subset of the OpenStack Identity API.
• The v4 API contains Keystone-NG capabilities that are not provided by Python Keystone.
• The generated OpenAPI document is the source of truth for routes and schemas implemented by the current build.

Do not assume that an API described by Python Keystone exists in Keystone-NG. Check the generated specification or the user API guide.

Parallel Deployment

Python and Rust Keystone can use the same deployment while a reverse proxy routes requests according to path ownership. Each implementation manages its own schema additions; database changes must not interfere with the other service. See Installation for the deployment model.

Fernet interoperability requires every node to use the same key repository. See the administrator Fernet guide.

Performance Comparison

The repository contains a Goose-based load-test crate that exercises selected API paths while recording latency and throughput. Pull-request benchmark workflows use it to detect regressions; results depend on the selected scenario, database, machine, concurrency, and build profile.

Treat benchmark artifacts from the current CI run as authoritative. Do not compare an isolated historical number with a different deployment or workload. The benchmark workflows under .github/workflows/benchmark*.yml record the commands and environment used for each result.

User Documentation

This section is for people and applications consuming Keystone-NG APIs, including delegated domain managers.

• API guide covers version discovery, OpenAPI, and common resource requests.
• Authentication covers password, TOTP, passkeys, and token headers.
• Feature guides describe the client side of Fernet tokens, federation, OAuth2/OIDC, SCIM and its API keys, Kubernetes authentication, and identity mapping.

Each feature guide links to its administrator counterpart when deployment or configuration work is required.

API Guide

Keystone exposes version discovery at /, /v3, and /v4.

• v3 contains the implemented OpenStack Identity resources, including tokens, domains, projects, users, groups, roles, assignments, catalog resources, credentials, and EC2 token validation.
• v4 contains Keystone-NG features such as passkeys, federation, Kubernetes authentication, mapping rulesets, OAuth2/OIDC, SCIM, and token restrictions.

The deployment’s reverse proxy determines whether Python Keystone or Keystone-NG handles a request. See Compatibility.

OpenAPI and Swagger UI

• Swagger UI provides the rendered API reference.
• OpenAPI YAML is the generated specification.
• A running server exposes Swagger UI at /swagger-ui/ and JSON at /api-docs/openapi.json.

The generated specification is authoritative for request fields, response fields, and status codes.

Common Resource Requests

The examples use an administrator token stored in TOKEN.

Projects

curl -X POST "${KEYSTONE_URL}/v3/projects" \
  -H "X-Auth-Token: ${TOKEN}" \
  -H 'Content-Type: application/json' \
  -d '{
    "project": {
      "name": "service",
      "domain_id": "default",
      "enabled": true,
      "is_domain": false
    }
  }'

A successful create returns 201 Created. The remaining lifecycle operations use GET /v3/projects, GET, PATCH, and DELETE on /v3/projects/{project_id}.

Users

curl -X POST "${KEYSTONE_URL}/v3/users" \
  -H "X-Auth-Token: ${TOKEN}" \
  -H 'Content-Type: application/json' \
  -d '{
    "user": {
      "name": "service-user",
      "domain_id": "default",
      "enabled": true,
      "password": "<initial-password>"
    }
  }'

The response never returns the supplied password. Use GET /v3/users and the GET, PATCH, and DELETE operations on /v3/users/{user_id} for the rest of the lifecycle.

Credentials

Credential blob is a JSON-encoded string, not a nested JSON object:

curl -X POST "${KEYSTONE_URL}/v3/credentials" \
  -H "X-Auth-Token: ${TOKEN}" \
  -H 'Content-Type: application/json' \
  -d '{
    "credential": {
      "type": "totp",
      "blob": "{\"seed\":\"<base32-seed>\"}"
    }
  }'

Credential responses can contain decrypted secret material. Treat the complete response as sensitive.

Authentication

Keystone supports multiple authentication methods. Password and TOTP use the v3 token API, while passkeys use the v4 WebAuthn flow.

Password

Password authentication uses POST /v3/auth/tokens:

curl -i -X POST "${KEYSTONE_URL}/v3/auth/tokens" \
  -H 'Content-Type: application/json' \
  -d '{
    "auth": {
      "identity": {
        "methods": ["password"],
        "password": {
          "user": {
            "name": "admin",
            "domain": {"id": "default"},
            "password": "<password>"
          }
        }
      },
      "scope": {
        "project": {
          "name": "admin",
          "domain": {"id": "default"}
        }
      }
    }
  }'

Success returns 201 Created; the token secret is in the X-Subject-Token response header.

TOTP

TOTP uses the same endpoint with method totp. The user supplies an existing user ID and the current passcode:

{
  "auth": {
    "identity": {
      "methods": ["totp"],
      "totp": {
        "user": {"id": "<user_id>", "passcode": "123456"}
      }
    }
  }
}

Using and Managing Tokens

Send a token as X-Auth-Token when calling authenticated APIs. To validate a token, send it as X-Subject-Token to GET /v3/auth/tokens; to revoke it, send the same headers to DELETE /v3/auth/tokens.

Other authentication mechanisms have dedicated feature guides. Direct application-credential authentication is not currently dispatched by the v3 token handler and is therefore not documented as supported.

Passkey Authentication (WebAuthn)

Keystone-NG supports registration of a hardware security device and a two-step passkey authentication ceremony. Platform passkeys that require an embedded browser are not currently supported by the client flow described here.

sequenceDiagram
    participant Authenticator
    Client->>Server: Authentication request
    Server->>Client: Challenge to be signed
    Client->>Authenticator: Challenge
    Authenticator->>+Authenticator: Sign with the private key and verify user presence
    Authenticator->>Client: Signed Challenge
    Client->>Server: Signed Challenge
    Server->>Server: Verify signature
    Server->>Client: Token

Registration

Registration uses these v4 endpoints for the target user:

1. POST /v4/users/{user_id}/passkeys/register_start
2. POST /v4/users/{user_id}/passkeys/register_finish

The first response contains the WebAuthn creation options. Pass those options to the authenticator and send its signed result to the finish endpoint.

Authentication

Authentication uses:

1. POST /v4/auth/passkey/start
2. POST /v4/auth/passkey/finish

The start response contains a challenge. The authenticator verifies user presence and signs that challenge; the finish request returns a Keystone token after successful verification.

See the administrator guide for relying-party configuration and deployment requirements.

Fernet Tokens for API Consumers

Fernet is Keystone-NG’s default v3 token format. Clients treat the value as an opaque bearer credential: do not parse it, log it, or depend on its encoded shape.

Use the token from the X-Subject-Token response header as X-Auth-Token on later requests. Validation and revocation use the v3 token operations described in Authentication.

Tokens expire according to deployment policy and may stop working after revocation or key-retention changes. Obtain a new token instead of persisting a Fernet token as a long-lived application secret.

Operators should use the Fernet administrator guide for key setup and rotation.

Federation

Federation exchanges an identity established by an external provider for a Keystone token. Mapping rules resolve external claims into a Keystone principal, scope, roles, and groups.

• Authorization Code flow covers browser-assisted OIDC login.
• JWT authentication covers exchanging an existing provider JWT.

Provider registration, allowed redirect URIs, mapping rules, and vendor setup belong to the federation administrator guide.

Authentication Using the Authorization Code Flow

sequenceDiagram

    Actor Human
    Human ->> Cli: Initiate auth
    Cli ->> Keystone: Fetch the OP auth url
    Keystone --> Keystone: Initialize authorization request
    Keystone ->> Cli: Returns authURL of the IdP with cli as redirect_uri
    Cli ->> User-Agent: Go to authURL
    User-Agent -->> IdP: opens authURL
    IdP -->> User-Agent: Ask for consent
    Human -->> User-Agent: give consent
    User-Agent -->> IdP: Proceed
    IdP ->> Cli: callback with Authorization code
    Cli ->> Keystone: Exchange Authorization code for Keystone token
    Keystone ->> IdP: Exchange Authorization code for Access token
    IdP ->> Keystone: Return Access token
    Keystone ->> Cli: return Keystone token
    Cli ->> Human: Authorized

TLDR

The user client (cli) sends authentication request to Keystone specifying the identity provider and optionally the scope (no credentials in the request). In the response the user client receives the time limited URL of the IDP that the user must open in the browser. When authentication in the browser is completed the user is redirected to the callback that the user also sent in the initial request (most likely on the localhost). User client is catching this callback containing the OIDC authorization code. Afterwards this code is being sent to the Keystone together with the authentication state and the user receives regular scoped or unscoped Keystone token.

Identity provider and mapping configuration

The identity provider is bound to a domain via --domain-id and references a mapping ruleset through --default-mapping-name. The mapping ruleset (managed at /v4/mappings/rulesets) defines how JWT/OIDC claims are mapped to Keystone identities. The IDP --default-mapping-name must match the mapping_id or rule_name in the ruleset so that the engine can resolve the correct mapping at callback time.

Note: Legacy federation mappings (/v4/federation/mappings) have been replaced by the unified mapping engine (/v4/mappings/rulesets).

User domain mapping

A Keystone identity provider can be bound to a single domain by setting the domain-id attribute on it. This means all users federated from such IDP would be placed in the specified domain.

Domain resolution can also be controlled by the mapping ruleset through its domain_resolution_mode:

• Fixed: locked to the IDP domain
• ClaimsOrMapping: rules may override domain via claims templates
• ClaimsOnly: neither IDP nor mapping is bound to a domain

The ultimate flexibility of having a single IdP for multiple domains is by using the user_domain_id template in the mapping rule to specify domain the user should belong to. Authentication with the claim missing is going to be rejected.

User group membership

When a user authenticates using OIDC, group memberships are synced on every login via the mapping engine. The mapping ruleset defines which groups the user should be assigned to based on JWT/OIDC claims. With IdentityMode::Local the engine performs user create/find and group membership sync on every login.

The major consequence when using application credentials that rely on roles assigned through group memberships is that the user needs to periodically login using the OIDC, since only the mapping engine can refresh group memberships.

Authenticating with JWT

It is possible to authenticate with the JWT token issued by the federated IdP. More precisely it is possible to exchange a valid JWT for the Keystone token. There are few different use scenarios that are covered.

Since the JWT was issued without any knowledge of the Keystone scopes it becomes hard to control scope. In the case of real human login the Keystone may issue unscoped token allowing user to further rescope it. In the case of the workflow federation that introduces a potential security vulnerability. As such in this scenario the mapping ruleset is responsible to fix the scope.

Login request looks following:

  curl https://keystone/v4/federation/identity_providers/${IDP}/jwt -X POST -H "Authorization: bearer ${JWT}" -H "openstack-mapping: ${MAPPING_RULESET_NAME}"

The openstack-mapping header references a mapping ruleset managed at /v4/mappings/rulesets. The engine resolves the ruleset from the IDP and uses the header as the rule_name hint for targeted rule matching.

Regular user obtains JWT (ID token) at the IdP and presents it to Keystone

In this scenario a real user (human) is obtaining the valid JWT from the IDP using any available method without any communication with Keystone. This may use authorization code grant, password grant, device grant or any other enabled method. This JWT is then presented to the Keystone and an explicitly requested mapping ruleset converts the JWT claims to the Keystone internal representation after verifying the JWT signature, expiration and further restricted bound claims.

Workload federation

Automated workflows (Zuul job, GitHub workflows, GitLab CI, etc) are typical workloads not being bound to any specific user and are more regularly considered being triggered by certain services. Such workflows are usually in possession of a JWT token issued by the service owned IdP. Keystone allows exchange of such tokens to the regular Keystone token after validating token issuer signature, expiration and applying the configured mapping ruleset. Since in such case there is no real human the mapping also needs to be configured slightly different.

• It is strongly advised the mapping ruleset must fill token_user_id, token_project_id via the Authorization fields. This allows strong control of which technical account is being used and which project such request can access.

• Mapping ruleset should use bound_audiences, bound_claims, bound_subject, etc to control the tokens issued by which workflows are allowed to access OpenStack resources.

GitHub workflow federation

In order for the GitHub workflow to be able to access OpenStack resources it is necessary to register GitHub as a federated IdP and establish a corresponding attribute mapping of the jwt type.

IdP:

"identity_provider": {
    "name": "github",
    "bound_issuer": "https://token.actions.githubusercontent.com",
    "jwks_url": "https://token.actions.githubusercontent.com/.well-known/jwks"
}

Mapping:

"mapping": {
   "type": "jwt",
   "name": "gtema_keystone_main",
   "idp_id": <IDP_ID>,
   "domain_id": <DOMAIN_ID>,
   "bound_audiences": ["https://github.com"],
   "bound_subject": "repo:gtema/keystone:pull_request",
   "bound_claims": {
       "base_ref": "main"
   },
   "user_id_claim": "actor_id",
   "user_name_claim": "actor",
   "token_user_id": <UID>
}

TODO: add more claims according to docs

A way for the workflow to obtain the JWT is described here.

...
permissions:
  token: write
  contents: read

job:
  ...
  - name: Get GitHub JWT token
    id: get_token
    run: |
      TOKEN_JSON=$(curl -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \
      "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=https://github.com")

      TOKEN=$(echo $TOKEN_JSON | jq -r .value)
      echo "token=$TOKEN" >> $GITHUB_OUTPUT
  ...
  # TODO: build a proper command for capturing the actual token and/or write a dedicated action for that.
  - name: Exchange GitHub JWT for Keystone token
    run: |
      KEYSTONE_TOKEN=$(curl -H "Authorization: bearer ${{ steps.get_token.outputs.token }}" -H "openstack-mapping: gtmema_keystone_main" https://keystone_url/v4/federation/identity_providers/IDP/jwt)

OAuth2 / OIDC Provider — User & Application Guide

Keystone can act as a standards-compliant OAuth2 Authorization Server / OpenID Connect Provider (OP). This means:

• Human users can log in through a browser (Authorization Code + PKCE, or the Device Authorization Grant for CLIs/headless machines) and get a short-lived, self-contained JWT instead of a Fernet token.
• Automated workloads (CI/CD pipelines, Kubernetes controllers, service accounts) can authenticate with client_credentials and call OpenStack APIs directly with the resulting JWT — no Fernet exchange needed.
• Third-party applications (Grafana, Harbor, internal portals) can use Keystone as a normal OIDC identity provider (“Login with OpenStack”).

See ADR 0026 for the full design. This page covers the flows you actually call. If you’re operating/deploying the provider rather than consuming it, see the administrator guide.

All endpoints below are under /v4/oauth2/{domain_id}/... — the OP is per-domain, so domain_id is always part of the path, and each domain has its own issuer and signing keys.

Discovery

GET /v4/oauth2/{domain_id}/.well-known/openid-configuration
GET /v4/oauth2/{domain_id}/jwks

Both are unauthenticated. Point any standard OIDC library at the discovery document and it will find authorization_endpoint, token_endpoint, jwks_uri, revocation_endpoint, introspection_endpoint, userinfo_endpoint, supported grant types, and scopes.

Scopes

• openid, profile, email — standard OIDC identity scopes.
• openstack:api — a distinct, explicit scope. Only when this is requested and granted does the returned access_token carry OpenStack authorization data (openstack_context: scope + effective roles) and an aud that OpenStack services will accept (openstack-apis). Without it, you get a minimal identity token good only for calling Keystone’s own /userinfo — not usable against Nova/Neutron/etc. Not yet supported on the authorization_code and device_code grants: requesting it there is rejected with invalid_scope. Use client_credentials or token exchange to obtain an OpenStack-capable token.
• Omitting scope entirely defaults to the client’s full allowed_scopes — except openstack:api is never implied by omission; you must request it explicitly every time.

Machine-to-machine: client_credentials

For CI/CD, Kubernetes operators, Terraform controllers, and any workload holding a registered client secret.

POST /v4/oauth2/{domain_id}/token
Content-Type: application/x-www-form-urlencoded

grant_type=client_credentials&client_id=<id>&client_secret=<secret>&scope=openstack:api

Response:

{
  "access_token": "eyJ...",
  "token_type": "Bearer",
  "expires_in": 900,
  "scope": "openstack:api"
}

Use access_token directly as Authorization: Bearer <token> against any OpenStack service running the native JWT middleware. No id_token is issued for this grant.

Human login: Authorization Code + PKCE

For browser-based apps and CLIs that can open a browser.

1. Redirect the user to:

   GET /v4/oauth2/{domain_id}/authorize
     ?response_type=code
     &client_id=<id>
     &redirect_uri=<your callback>
     &scope=openid profile openstack:api
     &state=<random>
     &code_challenge=<S256 PKCE challenge>
     &code_challenge_method=S256
   

   PKCE (S256 only) is mandatory for public clients. Keystone serves its own login and consent pages.

2. On success, your redirect_uri receives ?code=...&state=.... Exchange the code:

   POST /v4/oauth2/{domain_id}/token
   Content-Type: application/x-www-form-urlencoded
   
   grant_type=authorization_code&code=<code>&redirect_uri=<same as above>
   &code_verifier=<PKCE verifier>&client_id=<id>[&client_secret=<secret>]
   

   Response includes access_token, id_token, expires_in, and (if the client is registered for refresh_token) a refresh_token.

3. Refresh when the access token expires:

   grant_type=refresh_token&refresh_token=<token>&client_id=<id>
   

   Refresh tokens rotate on every use (a new one is returned each time; the old one becomes invalid). Do not reuse an old refresh token — presenting an already-used one is treated as a possible theft and revokes the entire token family, forcing a fresh login.

   A refresh token family also has an absolute lifetime (default 90 days from the original login, [oauth2] refresh_token_absolute_lifetime_days). Rotating never extends it: once reached, the refresh fails with invalid_grant and the user must log in again. On every refresh the server re-checks that the user still exists, is enabled and that the domain is enabled; otherwise the whole family is revoked and invalid_grant is returned. The expires_in of the response is the plain access-token lifetime. Once roles are embedded in access tokens, a refreshed token carries the roles resolved at rotation time.

CLI / headless login: Device Authorization Grant (RFC 8628)

For openstack/osc CLI and other headless clients, the same flow every major cloud CLI (aws sso, gcloud, az) uses.

1. Start the flow:

   POST /v4/oauth2/{domain_id}/device_authorization
   Content-Type: application/x-www-form-urlencoded
   
   client_id=<id>&scope=openid profile openstack:api
   

   Response:

   {
     "device_code": "...",
     "user_code": "WDJB-MJHT",
     "verification_uri": "https://keystone.example.com/v4/oauth2/<domain_id>/device",
     "verification_uri_complete": "https://keystone.example.com/v4/oauth2/<domain_id>/device?user_code=WDJB-MJHT",
     "expires_in": 600,
     "interval": 5
   }
   

2. Show the user verification_uri_complete (or verification_uri + user_code) and have them approve it in a browser.

3. Poll for the token:

   grant_type=urn:ietf:params:oauth:grant-type:device_code
   &device_code=<device_code>&client_id=<id>
   

   Poll no faster than interval seconds — polling too fast returns slow_down per RFC 8628 §3.5, which means “back off further,” not a hard failure. user_code uses an unambiguous character set ([A-Z0-9] minus O/0/I/l/1) so it’s easy to type by hand.

Managing connected applications

When you approve an application and tick “Remember my decision for this application”, it is listed as a connected application and the consent page is skipped next time it asks for the same or fewer scopes. Asking for an additional scope shows the page again, as does prompt=consent.

List the applications you approved:

$ curl -H "X-Auth-Token: $TOKEN" \
    $KEYSTONE/v4/users/$USER_ID/oauth2/consents
{"consents": [{"client_id": "...", "client_name": "My App",
  "scopes": ["openid", "profile"], "authorization_target": null,
  "granted_at": 1790000000, "updated_at": 1790000000}]}

Withdraw one:

$ curl -X DELETE -H "X-Auth-Token: $TOKEN" \
    $KEYSTONE/v4/users/$USER_ID/oauth2/consents/$CLIENT_ID

This forgets the approval and revokes the application’s refresh tokens for your account, so it cannot renew its access; access tokens it already holds work until they expire. You can manage your own consents with an ordinary token, but not with an application credential or a trust token; an administrator can manage anyone’s. The endpoint answers 404 when there is neither a remembered consent nor a live refresh token for that application.

Token types you’ll see

• id_token — identity only, aud is your client_id. Never carries roles or OpenStack scope; only your client’s configured claims_template output is added.
• access_token (openstack:api granted on a supported grant, or any client_credentials grant) — carries openstack_context (scope + effective roles at issuance time) and aud: "openstack-apis", valid against every OpenStack service that trusts the token’s issuer (the token’s domain). On the token-exchange grant you can narrow it with the RFC 8693 audience parameter (repeatable, one OpenStack service type per value, e.g. audience=compute), which yields aud: ["openstack-apis:compute", ...] accepted only by those service types. An unknown service type is rejected with invalid_target. This is what OpenStack services accept.
• access_token (openstack:api not granted) — minimal, aud is your own client_id, not usable against any OpenStack service. Good only for /userinfo (and for revocation).

Access and ID tokens are short-lived (15 minutes by default) and stateless bearer tokens. You can end a session early with the revocation endpoint below (refresh tokens always; access tokens individually); otherwise wait out exp (or an operator triggers emergency signing-key rotation, which is out of your hands as a client). Treat them like any other bearer credential: don’t log them, don’t put them in URLs.

UserInfo endpoint

GET or POST /v4/oauth2/{domain_id}/userinfo (OIDC Core §5.3), advertised as userinfo_endpoint in the discovery document. Authenticate with the OIDC access_token as an RFC 6750 bearer credential:

curl -H "Authorization: Bearer $ACCESS_TOKEN" \
  https://keystone.example.com/v4/oauth2/$DOMAIN_ID/userinfo

The token must carry the openid scope, be unexpired and not revoked, and its user and client must still exist and be enabled. The response is plain JSON (userinfo_signing_alg_values_supported is ["none"]) with Cache-Control: no-store:

• sub — always.
• name, preferred_username, updated_at — with the profile scope.
• email, email_verified — with the email scope, and only when the user has an email address. Keystone does not verify addresses, so email_verified is false unless the user record explicitly says otherwise.
• Your client’s claims_template output, identical to the id_token.

Failures follow RFC 6750 §3: 401 with WWW-Authenticate: Bearer (no token) or Bearer error="invalid_token" (bad, expired, revoked or foreign token, disabled user or client), and 403 with error="insufficient_scope" when the token lacks openid. Access tokens issued with openstack:api (or by client_credentials) are not accepted here. The endpoint is rate limited per source IP.

Revoking tokens (RFC 7009)

POST /v4/oauth2/{domain_id}/revoke
Content-Type: application/x-www-form-urlencoded

token=<refresh or access token>&token_type_hint=refresh_token

Authenticate the client exactly as for /token (HTTP Basic, or client_id / client_secret in the body). token_type_hint (refresh_token or access_token) is optional and only affects lookup order.

• Refresh token — the whole refresh family (the token and everything rotated from it) is revoked; any later refresh_token grant fails with invalid_grant.
• Access token — its jti is added to the domain’s JTI revocation list (/jwks/revocation) until the token’s own exp. Services that enforce the list (see the admin guide) reject it from then on, within the list’s cache TTL (60s). If the token carries a sid claim (the id of the refresh family that minted it), that refresh family is revoked too, ending the session. ID tokens carry no jti and are ignored.

The endpoint always answers 200 with an empty body for an authenticated, well-formed request — including unknown, expired, already-revoked tokens and tokens that belong to another client — so it never reveals whether a token exists. A token is only revoked if it belongs to the authenticated client. Bad client credentials return 401 invalid_client, a missing token returns 400 invalid_request, and the endpoint is rate limited like /token.

Limitation: revocation is one-directional. Revoking an access token also ends its refresh family (via sid), but revoking a refresh token does not revoke access tokens already issued from it; they expire on their own (or revoke them individually). Access tokens from client_credentials / token-exchange have no refresh family and no sid.

Token introspection (RFC 7662)

POST /v4/oauth2/{domain_id}/introspect
Content-Type: application/x-www-form-urlencoded

token=<refresh or access token>&token_type_hint=access_token

Advertised as introspection_endpoint in the discovery document. Use it when a service cannot accept the stateless revocation window of access tokens: unlike offline verification it sees the domain’s JTI revocation list and the refresh-token store as they are now.

Only an authenticated confidential client of the same domain may call it. Authenticate exactly as for /token (HTTP Basic, or client_id / client_secret in the body); public clients get 401 invalid_client. token_type_hint (refresh_token or access_token) is optional and only affects lookup order.

The answer is always 200 with a JSON body and Cache-Control: no-store. A token that is unknown, malformed, expired, revoked, already used (refresh) or issued for another domain yields exactly {"active": false}; the response never says why. For an active token active is true and the body carries:

• scope, client_id, sub, exp, iat, token_use. For a refresh token exp is bounded by the refresh family’s lifetime cap.
• Access tokens additionally: token_type (Bearer), nbf, aud, iss, jti. OpenStack access tokens (openstack:api, client_credentials) also include openstack_context and delegation_context, the scope and effective roles the token will act with.

A missing token returns 400 invalid_request, bad client credentials return 401 invalid_client, and the endpoint is rate limited per source IP and per client.

Errors

Token endpoint errors follow RFC 6749 §5.2:

{ "error": "invalid_grant", "error_description": "..." }

Common ones: invalid_client (bad client_id/secret), invalid_grant (expired/used code, revoked refresh token, wrong PKCE verifier), invalid_scope (requested a scope outside allowed_scopes — the server never silently narrows a request), slow_down / authorization_pending (device flow polling).

SCIM v2 Support

This page is a protocol-level reference for whoever configures the SCIM side of an Identity Provider connector (Okta, Entra ID, Workday, …) against Keystone, and a compatibility matrix against RFC 7644 for anyone evaluating whether Keystone’s SCIM implementation fits their IdP’s requirements. For how to register a realm and grant it access, see the Administrator Guide. For full design rationale, see ADR 0024.

Keystone implements a deliberately restricted subset of RFC 7644, not full compliance — the restrictions exist to bound worst-case query/PATCH complexity per request (the same posture applied elsewhere in this codebase to claim mapping and rate limiting). Most enterprise IdPs, including Okta and Entra ID, tolerate a narrower filter grammar and bulk.supported: false without issue; check the matrix below against your specific connector before assuming a feature works.

Base URL & Authentication

https://<host>/SCIM/v2/{domain_id}/...
Authorization: Bearer kscim_...

Every request is scoped to one domain and authenticates with a bearer API key (see API-Key Authentication) — never a Fernet token. The key must belong to an active, registered realm for that (domain_id, provider_id) coordinate, or every request gets 403 regardless of role (see Realm Activation Gate).

Content Negotiation

Request bodies (POST/PUT/PATCH carrying a payload) must declare Content-Type: application/scim+json or, for connectors that only speak plain JSON, application/json — either is accepted. Anything else, or a missing header on a request that does carry a body, is rejected with 415 Unsupported Media Type. Every response carries Content-Type: application/scim+json, including error responses.

Discovery

GET /SCIM/v2/{domain_id}/ServiceProviderConfig
GET /SCIM/v2/{domain_id}/Schemas
GET /SCIM/v2/{domain_id}/ResourceTypes

Unauthenticated within the SCIM sub-router (bearer auth is still accepted, just not required) — most connectors probe these before presenting credentials. ServiceProviderConfig honestly advertises what’s not supported rather than claiming full compliance:

{
  "patch": { "supported": true },
  "bulk": { "supported": false, "maxOperations": 0, "maxPayloadSize": 0 },
  "filter": { "supported": true, "maxResults": 200 },
  "changePassword": { "supported": false },
  "sort": { "supported": false },
  "etag": { "supported": true },
  "authenticationSchemes": [{ "type": "oauthbearertoken", "primary": true }]
}

Resource Endpoints

POST   /SCIM/v2/{domain_id}/Users            GET   .../Users        GET .../Users/{id}
PUT    /SCIM/v2/{domain_id}/Users/{id}        PATCH .../Users/{id}   DELETE .../Users/{id}

POST   /SCIM/v2/{domain_id}/Groups            GET   .../Groups       GET .../Groups/{id}
PUT    /SCIM/v2/{domain_id}/Groups/{id}       PATCH .../Groups/{id}  DELETE .../Groups/{id}

An HTTP method not mapped for a given path (e.g. POST .../Users/{id}, or any method on /ServiceProviderConfig other than GET) returns 405 Method Not Allowed.

Ownership fencing

A resource is only visible to the realm that created it. GET/PUT/PATCH/ DELETE against an id owned by a different realm — or a same-ID resource that doesn’t exist at all — both return an identical 404, by design: this prevents realm-boundary probing via response-shape differences.

POST — required fields, schemas validation

The request body’s schemas array must contain the resource’s core schema URI (urn:ietf:params:scim:schemas:core:2.0:User / ...:Group) — a missing or mismatched schemas array is rejected with 400 invalidValue. For Users, externalId is additionally mandatory (400 if empty/absent) and drives deterministic id derivation so a later federated JIT login for the same IdP sub claim converges onto the same account rather than creating a duplicate.

Response meta

Every response carries meta.location (an absolute URL), and 201 Created responses on POST additionally carry an HTTP Location header matching it. RFC 7644 doesn’t mandate the header outside 201; the body field is present on every response regardless.

Attribute Mapping (User)

Attribute Mapping (Group)

A members entry referencing a user owned by a different realm, or a manually-created user with no SCIM ownership record at all, is rejected with 400 invalidValue on both PUT and PATCH add.

DELETE semantics

Neither Users nor Groups are hard-deleted by DELETE. A User is disabled and its sessions revoked; a Group has its role assignments immediately stripped (closing live authorization) while its membership snapshot is retained for a forensic window. Both become invisible to subsequent GET/PUT/PATCH/List (404) immediately. See Deprovisioning & retention for the retention window and operator purge-now path — this deviates from a strict reading of RFC 7644 §3.6, which doesn’t distinguish soft- from hard-delete.

Filtering

filter := term (LOGICAL_OP term)*      # "and"/"or" MUST NOT be mixed in one filter string
term    := ATTR OP value
OP      := eq | ne | co | sw | pr

No nested/parenthesized expressions, no complex-attribute filters (emails[type eq "work"]). A filter string over 512 bytes or 8 terms is rejected. Violations return 400 invalidFilter.

Pagination

startIndex (1-based, default 1), count (default and max 200). No cursor/continuation token — a bounded scan over the realm’s own resource set, excluding deprovisioned entries.

PATCH

Operations: [{op, path, value}], add/replace/remove only, restricted to these top-level scalar paths:

id and meta are real SCIM attributes but always immutable — a PATCH naming either returns 400 mutability (distinct from a path that isn’t a recognized attribute at all, which returns 400 invalidPath). Any other unrecognized path, an array-index path, or a complex filter expression (emails[type eq "work"].value) also returns 400 invalidPath. PUT performs a full declarative replace, including a full membership resync for Groups.

ETags & Concurrency

GET/PUT/PATCH/POST responses carry a weak ETag: W/"<version>". Send If-Match: W/"<version>" on PUT/PATCH to get an atomic compare-and-swap; a stale version returns 412 Precondition Failed (no response body). This closes the lost-update race for concurrent push-group syncs from a single IdP without needing a distributed lock.

/Bulk and /Me

Both return 501 Not Implemented with a SCIM-shaped error body rather than a generic 404 — RFC 7644 clients commonly probe these before falling back. Neither is planned for this subset (see Explicitly out of scope below).

Errors

Standard envelope:

{
  "schemas": ["urn:ietf:params:scim:api:messages:2.0:Error"],
  "status": "409",
  "scimType": "uniqueness",
  "detail": "userName already exists within this domain"
}

noTarget, tooMany, invalidVers, and sensitive are defined by RFC 7644 §3.12 but never emitted — see the compatibility matrix below for why.

Explicitly out of scope

/Bulk, sortBy/sortOrder, arbitrary filter/PATCH path expressions (emails[type eq "work"]), and multi-valued complex-attribute addressing. Extending any of these requires a ratifying ADR 0024 revision given their DoS/complexity surface.

RFC 7644 Compatibility Matrix

API-Key Authentication

SCIM API keys are domain-owned machine credentials accepted only by the SCIM router. They are not substitutes for v3 Fernet tokens and are rejected by core v3 and v4 endpoints.

Send the key as a bearer credential:

Authorization: Bearer kscim_<secret>_<checksum>

The secret is shown once when an administrator creates or rotates the key. A client must store it as sensitive material and replace it when the administrator rotates or revokes the key.

See SCIM v2 Support for supported protocol behavior and the administrator guide for lifecycle and incident procedures.

Kubernetes TokenReview Authentication

Kubernetes workloads can exchange a service-account JWT for a Keystone token. Keystone validates the JWT with the configured Kubernetes cluster, converts the returned service-account identity into mapping claims, and applies the first matching identity-mapping rule.

Ask the domain administrator for the Kubernetes authentication instance ID and the mapping rule requirements before configuring a workload. The operator must register the cluster and enable the instance first.

Authentication request

Send the service-account JWT to the instance-specific endpoint:

POST /v4/k8s_auth/instances/{instance_id}/auth
Content-Type: application/json

{
  "jwt": "<service-account-jwt>",
  "rule_name": "ci-pipeline-admin"
}

The client JWT is also used to authorize Keystone’s TokenReview request to the Kubernetes API. The service account therefore needs permission to create tokenreviews.authentication.k8s.io, normally through the Kubernetes system:auth-delegator role.

On success, Keystone returns 200 OK, places the encoded token in the X-Subject-Token response header, and returns the token document in the body.

Authentication flow

sequenceDiagram
    App->>Keystone: POST /v4/k8s_auth/instances/{id}/auth
    Keystone->>Keystone: Check JWT expiration
    Keystone->>Kubernetes: TokenReview using the client JWT
    Kubernetes->>Keystone: Authenticated service-account identity
    Keystone->>Keystone: Flatten claims and evaluate mapping rules
    Keystone->>Keystone: Upsert the shadow virtual user
    Keystone->>App: Keystone token in X-Subject-Token

The TokenReview result produces these mapping claims:

The mapping rule determines the resulting principal, scope, roles, and groups. If rule_name is supplied, Keystone evaluates that rule first. If it does not match, normal first-match-wins evaluation continues.

The virtual user ID is deterministic for the service-account name, namespace, cluster, and Keystone deployment. Keystone also records the mapping ruleset version so a token is rejected if its authorization rules change later.

See Identity Mapping Rules and API for Kubernetes claim examples and rule management. See the administrator guide for cluster registration, CA trust, and operational requirements.

Identity Mapping Rules and API

Domain managers use the Identity Mapping Engine to resolve external identities from federation, Kubernetes, SPIFFE, and API clients into Keystone principals and security contexts. Instead of hardcoding authentication logic for each provider, Keystone evaluates ordered rules that the domain manager controls.

Deployment configuration and cluster-wide operational requirements are covered in the administrator guide.

Concept

The mapping engine decouples the authentication phase (verifying who the user is) from the authorization phase (determining what the user is in Keystone).

When an authentication request is received, the identity provider extracts “claims” (attributes about the identity) and passes them along with the source of the identity to the Mapping Engine. The engine evaluates these against a set of Rulesets. The first matching rule defines:

• The Principal (user or group) to be mapped to.
• The Scope and Role associations for the resulting token.

This allows administrators to dynamically change how external identities are mapped to internal users without changing code or reconfiguring the authentication providers.

Rule Structure

A MappingRuleSet contains an ordered list of MappingRules. The engine iterates through these rules in sequence and applies the first rule that satisfies its match criteria. Each rule consists of:

• Match Criteria: A nested boolean expression tree evaluating claims.
• Identity Binding: Defines the target Keystone user (user_name, user_id, user_domain_id) using optional template interpolation.
• Authorizations: Roles granted at the System, Domain, or Project level.
• Group Assignments: Groups the identity should be mapped to, with configurable resolution strategy.

Match Criteria

Match criteria define the boolean logic for evaluating conditions:

• AllOf: All conditions must match.
• AnyOf: At least one condition must match.
• AllOfStrict: All conditions must match, with an optional require_all_keys flag. When enabled, the engine verifies that all claim keys referenced in the criteria are present in the claims map. This prevents claim-suppression attacks where an attacker omits claims to bypass higher-priority rules.

Match Conditions

A match condition can be either:

• Condition: A leaf-level claim assertion (see Claim Conditions below).
• Nested: A sub-group of MatchCriteria, allowing complex boolean nesting.

Claim Conditions

A claim condition evaluates a specific claim key against a target value. Each condition must specify claim (the key to look up in the claims map):

• Equals: The claim value must equal the specified value. JSON primitives (numbers, booleans) are normalized to strings for comparison.
• AnyOf: The claim value must match at least one value in the values array.
• MatchesRegex: The claim value must match the given regex pattern. Regex patterns are cached with a 1024-entry LRU limit.

Identity Binding & Template Interpolation

The identity binding defines the resulting Keystone principal:

• user_name (required): String identifying the user. Supports template interpolation.
• user_id (optional): Explicit user identifier. Supports template interpolation.
• user_domain_id (optional): Domain context for the user. Supports template interpolation, subject to DomainResolutionMode constraints.
• is_system (default false): When true, grants system-level privileges. Rulesets containing system bindings are treated as immutable system mappings after creation.

Template Interpolation

Identity fields support two template tokens:

• ${claims.<key>}: Replaces with the first value from the claims map for the given key. Unresolved keys leave the token intact.
• ${enclosing_domain_id}: Replaces with the ruleset’s enclosing domain ID.

Templates are limited to 256 characters after resolution and cannot reference reserved keys (e.g., enclosing_domain_id is blocked in ${claims.*} syntax to prevent context shadowing).

Authorizations

Roles are assigned at one of three scope levels:

• System: Grants roles at the system scope (requires is_system: true). Requires system_id (typically "all") and roles.
• Domain: Grants roles on a specific domain. Requires domain_id and roles.
• Project: Grants roles on a specific project. Requires project_id, project_domain_id, and roles.

Role references support both id and name lookups.

Group Assignments

Group assignments map the external identity to localized Keystone groups:

• group_id (required): Immutable group identifier.
• group_name (required): Group name, supports template interpolation.
• group_domain_id (optional): Domain containing the group.
• strategy (default CreateOrGet):
  • CreateOrGet: Creates the group if it does not exist, or retrieves the existing one.
  • Get: Only retrieves an existing group; fails if the group is missing.

Domain Resolution Modes

Each ruleset specifies how the domain context for the resolved principal is determined:

• Fixed (default): Domain is fixed to the domain_id on the ruleset. user_domain_id templates cannot reference ${claims.*} tokens.
• ClaimsOnly: Domain is resolved exclusively from claims. At least one rule in the ruleset must use a ${claims.*} template in user_domain_id.
• ClaimsOrMapping: Domain is resolved from user_domain_id template first, falling back to the ruleset domain_id. Both claim templates and static values are permitted.

Authentication Providers

SPIFFE

SPIFFE (Secure Production Identity Framework for Everyone) provides a way to identify workloads across heterogeneous environments using SVIDs (SPIFFE Verifiable Identity Documents), typically x509 certificates.

Concept

SPIFFE authentication extracts the SVID from the mTLS x509 certificate and flattens it into a claims map for the mapping engine. The source type is spiffe, identified by the trust_domain field.

Claims Contract

The following claims are produced for SPIFFE identities:

• spiffe.id: Full SPIFFE ID URI (e.g., spiffe://example.org/spiffe/test-workload).
• spiffe.trust_domain: Trust domain from the SPIFFE ID (e.g., example.org).

Mapping Rule Examples

Example 1: Map specific SPIFFE ID to a System Admin

Grants system-level privileges to a specific administrative workload.

{
  "name": "system-admin-workload",
  "match": {
    "all_of": [
      {
        "type": "condition",
        "equals": {
          "claim": "spiffe.id",
          "value": "spiffe://example.org/spiffe/admin-tool"
        }
      }
    ]
  },
  "identity": {
    "user_name": "admin-workload",
    "is_system": true
  },
  "authorizations": [
    {
      "type": "system",
      "system_id": "all",
      "roles": [{ "id": "admin", "name": "admin" }]
    }
  ],
  "groups": []
}

Example 2: Map Trust Domain to a Project Role

Broadly maps any workload from a specific trust domain into a project with a specific role.

{
  "name": "trust-domain-mapping",
  "match": {
    "all_of": [
      {
        "type": "condition",
        "equals": {
          "claim": "spiffe.trust_domain",
          "value": "example.org"
        }
      }
    ]
  },
  "identity": {
    "user_name": "external-workload-user"
  },
  "authorizations": [
    {
      "type": "project",
      "project_domain_id": "default",
      "project_id": "project-123",
      "roles": [{ "id": "member", "name": "member" }]
    }
  ],
  "groups": [
    {
      "group_id": "external-group",
      "group_name": "External Workloads",
      "strategy": { "type": "create_or_get" }
    }
  ]
}

Example 3: Regex-based workload pattern matching

Maps workloads matching a path pattern using regex.

{
  "name": "dev-workloads-regex",
  "match": {
    "all_of": [
      {
        "type": "condition",
        "matches_regex": {
          "claim": "spiffe.id",
          "regex": "^spiffe://example.org/dev/.+$"
        }
      }
    ]
  },
  "identity": {
    "user_name": "dev-workload-user"
  },
  "authorizations": [
    {
      "type": "project",
      "project_domain_id": "default",
      "project_id": "dev-project",
      "roles": [{ "id": "member", "name": "member" }]
    }
  ],
  "groups": []
}

Example 4: AllOfStrict with trust domain and specific path

Prevents claim-suppression by requiring both claims to be present.

{
  "name": "strict-namespace-mapping",
  "match": {
    "all_of_strict": {
      "conditions": [
        {
          "type": "condition",
          "equals": {
            "claim": "spiffe.trust_domain",
            "value": "example.org"
          }
        },
        {
          "type": "condition",
          "equals": {
            "claim": "spiffe.id",
            "value": "spiffe://example.org/production/api"
          }
        }
      ],
      "require_all_keys": true
    }
  },
  "identity": {
    "user_name": "production-api"
  },
  "authorizations": [
    {
      "type": "project",
      "project_domain_id": "default",
      "project_id": "prod-project",
      "roles": [{ "id": "member", "name": "member" }]
    }
  ],
  "groups": []
}

Ruleset workflow

After the operator registers the SPIFFE trust domain, create a mapping ruleset with source type spiffe and the appropriate trust_domain. Match rules against spiffe.id or spiffe.trust_domain claims. Operator-only SPIFFE and admin-interface configuration belongs in the administrator guide.

Federation

Federation-based identity sources (e.g., OAuth 2.0, OIDC, SAML) authenticate via an IdP and extract claims from the resulting security token.

Concept

The ingress adapter performs the cryptographic validation of the federation token (signature verification, CRL checks, remote TokenReview calls) and flattens the claims into a map. The mapping engine then resolves the identity using the same rule-based matching as other providers. The source type is federation, identified by the idp_id field.

Claims Contract

Claims are provider-dependent and vary by federation protocol. Common claims include:

• sub: Subject identifier (unique to the IdP).
• preferred_username: Human-readable username.
• groups: Group memberships from the IdP.
• email: Email address.

Mapping Rule Examples

Example 1: Map Federation User by Subject

Resolve a specific federation user by their sub claim.

{
  "name": "federation-subject-mapping",
  "match": {
    "all_of": [
      {
        "type": "condition",
        "equals": {
          "claim": "sub",
          "value": "federation-user-123"
        }
      }
    ]
  },
  "identity": {
    "user_name": "${claims.preferred_username}",
    "user_domain_id": "${enclosing_domain_id}"
  },
  "authorizations": [
    {
      "type": "project",
      "project_domain_id": "default",
      "project_id": "project-123",
      "roles": [{ "id": "member", "name": "member" }]
    }
  ],
  "groups": [
    {
      "group_id": "federation-users",
      "group_name": "Federation Users",
      "strategy": { "type": "create_or_get" }
    }
  ]
}

Example 2: Map Federation Group Membership

Resolve identities based on the groups claim.

{
  "name": "federation-group-mapping",
  "match": {
    "all_of": [
      {
        "type": "condition",
        "any_of": {
          "claim": "groups",
          "values": ["engineering", "operations"]
        }
      }
    ]
  },
  "identity": {
    "user_name": "${claims.preferred_username}"
  },
  "authorizations": [
    {
      "type": "project",
      "project_domain_id": "default",
      "project_id": "shared-project",
      "roles": [{ "id": "member", "name": "member" }]
    }
  ],
  "groups": []
}

Kubernetes

Kubernetes authentication validates service accounts via the Kubernetes API server’s TokenReview endpoint and delegates identity resolution to the unified mapping engine. For detailed ingress flow, API, and configuration, see Kubernetes TokenReview Authentication.

The source type is k8s, identified by the cluster_id field.

Authentication Flow

The K8s authenticator validates the JWT via TokenReview, flattens claims, and delegates to the mapping engine for identity resolution, shadow registry upsert, and authorization. The optional rule_name field allows the client to target a specific rule within the ruleset.

Claims Contract

The mapping-engine path produces the following claims from the TokenReview response and JWT payload:

• k8s.serviceaccount.name: Service account name (e.g., "build-runner").
• k8s.serviceaccount.namespace: Kubernetes namespace (e.g., "ci-pipeline").
• k8s.aud: JWT audience claim, if present in the token.

The unique workload ID invariant is <serviceaccount_name>:<serviceaccount_namespace>, used for deterministic virtual user ID derivation.

Mapping Rule Examples

Example 1: Map by namespace and service account name

Grants access to a specific CI/CD pipeline service account.

{
  "name": "ci-pipeline-admin",
  "match": {
    "all_of": [
      {
        "type": "condition",
        "equals": {
          "claim": "k8s.serviceaccount.namespace",
          "value": "ci-pipeline"
        }
      },
      {
        "type": "condition",
        "any_of": {
          "claim": "k8s.serviceaccount.name",
          "values": ["build-runner", "deploy-agent"]
        }
      }
    ]
  },
  "identity": {
    "user_name": "svc-k8s-${claims.k8s.serviceaccount.name}"
  },
  "authorizations": [
    {
      "type": "project",
      "project_domain_id": "default",
      "project_id": "infra-project",
      "roles": [{ "id": "admin", "name": "admin" }]
    }
  ],
  "groups": []
}

Example 2: Regex-based monitoring agents

Matches any Prometheus-related service account in the monitoring namespace.

{
  "name": "monitoring-reader",
  "match": {
    "all_of": [
      {
        "type": "condition",
        "equals": {
          "claim": "k8s.serviceaccount.namespace",
          "value": "monitoring"
        }
      },
      {
        "type": "condition",
        "matches_regex": {
          "claim": "k8s.serviceaccount.name",
          "regex": "^prometheus-.*$"
        }
      }
    ]
  },
  "identity": {
    "user_name": "svc-k8s-${claims.k8s.serviceaccount.name}"
  },
  "authorizations": [
    {
      "type": "project",
      "project_domain_id": "default",
      "project_id": "monitoring-project",
      "roles": [{ "id": "reader", "name": "reader" }]
    }
  ],
  "groups": [
    {
      "group_id": "k8s-monitoring-group",
      "group_name": "Monitoring-Agents",
      "strategy": { "type": "get" }
    }
  ]
}

Example 3: AllOfStrict for namespace-scoped binding

Prevents claim-suppression attacks by requiring both namespace and name claims to be present.

{
  "name": "strict-k8s-binding",
  "match": {
    "all_of_strict": {
      "conditions": [
        {
          "type": "condition",
          "equals": {
            "claim": "k8s.serviceaccount.namespace",
            "value": "openstack"
          }
        },
        {
          "type": "condition",
          "equals": {
            "claim": "k8s.serviceaccount.name",
            "value": "nova-compute"
          }
        }
      ],
      "require_all_keys": true
    }
  },
  "identity": {
    "user_name": "svc-nova-compute"
  },
  "authorizations": [
    {
      "type": "project",
      "project_domain_id": "default",
      "project_id": "compute-project",
      "roles": [{ "id": "admin", "name": "admin" }]
    }
  ],
  "groups": []
}

Mapping API

The mapping engine is managed via the Keystone API (v4).

RuleSet Management

Imperative Rule Mutation

To avoid replacing the entire rules array, the API supports atomic mutations:

• Insert: Add a rule at a specific position (before or after a named anchor rule).
• Update: Replace an existing rule by its name.
• Delete: Remove a rule by its name.

Create Request Example

{
  "mapping": {
    "mapping_id": "unique-ruleset-id",
    "domain_id": "default-domain",
    "source": {
      "type": "spiffe",
      "trust_domain": "example.org"
    },
    "domain_resolution_mode": {
      "type": "fixed"
    },
    "enabled": true,
    "rules": [
      {
        "name": "admin-workload",
        "match": {
          "all_of": [
            {
              "type": "condition",
              "equals": {
                "claim": "spiffe.id",
                "value": "spiffe://example.org/spiffe/admin-tool"
              }
            }
          ]
        },
        "identity": {
          "user_name": "admin-workload",
          "is_system": true
        },
        "authorizations": [
          {
            "type": "system",
            "system_id": "all",
            "roles": [{ "id": "admin", "name": "admin" }]
          }
        ],
        "groups": []
      }
    ]
  }
}

Administrator configuration

The operator must configure the cluster salt and enforce cluster-wide security limits before mapping-backed authentication can be used. See Identity Mapping Administration.

Administrator Guides

This section is for operators and delegated domain administrators deploying and managing Keystone-NG.

• Getting started covers initial bootstrap.
• Configuration explains deployment-oriented configuration.
• Operations covers health, metrics, upgrades, and recovery.
• Command-line tools documents the service and administration commands used by operators.
• Security collects production guardrails.
• API policy enforcement documents OPA integration.
• Fernet tokens and distributed storage provide dedicated runbooks.
• Feature guides cover passkeys, federation, OAuth2/OIDC, SCIM and its API keys, Kubernetes authentication, identity mapping, LDAP, dynamic auth plugins, the audit trail, and OpenTelemetry export.

Client requests and authentication flows belong in the user documentation. Implementation and test guidance belongs in the contributor documentation.

Getting Started as an Administrator

Install Keystone-NG and configure its database, OPA service, and interfaces before bootstrapping identity resources. See Installation and Configuration.

Bootstrap

Bootstrap requires [interface_admin] and creates or updates the default domain, admin project, admin user, standard roles, implied-role hierarchy, and system/project admin assignments. Catalog endpoints are added when their URL options are supplied.

OS_BOOTSTRAP_PASSWORD='<password>' \
  keystone-manage --config /etc/keystone/keystone.conf bootstrap

The username and project default to admin. Use keystone-manage bootstrap --help for the current username, project, region, and endpoint options.

After bootstrap, authenticate through POST /v3/auth/tokens as described in the user authentication guide and check /health and /ready on the metrics listener.

Administrator Configuration Guide

Keystone reads an OpenStack-style INI file. Values are merged in this order, with later sources winning:

1. The file selected with keystone --config.
2. The optional file named by KEYSTONE_SITE_VARS_FILE.
3. Environment variables prefixed with OS_, using __ between section and option.

For example, OS_API_POLICY__OPA_BASE_URL=http://opa:8181 overrides opa_base_url in [api_policy].

The configuration reference lists every section and directs operators to the authoritative detailed guide.

Interfaces

Use [interface_public] for the public HTTP listener. Terminate public HTTPS at a trusted reverse proxy or load balancer and restrict direct access to the listener. Public SPIFFE is not currently implemented.

The optional [interface_internal] listener supports SPIFFE mTLS and requires SPIFFE_ENDPOINT_SOCKET. [interface_admin] uses SPIFFE mTLS over a Unix-domain socket and can restrict peer UID and GID. [interface_metrics] exposes health, readiness, and Prometheus metrics and must not be placed on an untrusted network.

Core Services

• [database] selects the SQL connection.
• [api_policy] selects OPA and an optional local policy directory.
• [auth] enables built-in and registered authentication methods.
• [token] selects the token provider and lifetime.
• [fernet_tokens] configures the Fernet key repository.
• [distributed_storage] enables OpenRaft storage and its KEK and transport settings.

Feature-specific configuration belongs with the corresponding administrator guide. Do not copy partial distributed-storage examples into production; follow the complete storage runbook.

Operations

Health and Readiness

The dedicated [interface_metrics] listener defaults to 0.0.0.0:8099.

curl http://keystone:8099/health
curl http://keystone:8099/ready

Both endpoints report database, policy-engine, and distributed-storage status. /ready returns 503 Service Unavailable for degraded components; /health allows warning states to remain 200 OK.

With distributed storage enabled, a node is only ready when all of the following hold; otherwise the raft component reports warn with the reasons:

• the Raft cluster is initialized and a leader is known;
• when the node is the leader, a quorum acknowledged it within the last 6 seconds;
• the applied index trails the cluster commit index by at most 1000 entries (a node installing a snapshot or catching up after a partition, including one that has not applied a new DEK epoch yet, is not ready);
• no partition is quarantined on this node.

These states are not fatal, so they affect only readiness: use /health for liveness and startup probes. A Raft core that has stopped with a fatal error (storage failure, panic, shutdown) is the one exception — it is reported as error, which fails /health as well, so the liveness probe restarts the pod. In Kubernetes, run the storage StatefulSet with podManagementPolicy: Parallel, otherwise a full restart of a multi-node cluster waits for the first pod to become ready, which it cannot do without a quorum.

Metrics and Logging

GET /metrics returns Prometheus text format. Current exported metrics include audit event, drop, queue-depth, spool, sink and key-version series (keystone_audit_*, listed in ADR 0023 §Observability; ready-made alert rules are in deploy/prometheus/alert_rules.yaml) and dynamic-auth-plugin load failures, and HTTP, authentication, policy, token, rate-limit and cache series. The same data can be pushed over OTLP; see OpenTelemetry.

With distributed storage enabled, /metrics also exports the per-node keystone_raft_* series (alert rules in the keystone_raft group of the same file):

Keystone uses structured logging. Protect logs because request and identity metadata can be sensitive, and never enable logging that records bearer tokens, credentials, or decrypted policy input.

Upgrades

1. Back up the database, distributed storage, and key repositories.
2. Review configuration and migration changes.
3. Apply database migrations with keystone-manage db sync.
4. Upgrade one instance at a time where the deployment supports rolling replacement.
5. Verify health, readiness, token issuance, token validation, and representative authorized API calls before continuing.

Coordinate Fernet and distributed-encryption key changes separately from binary upgrades. Follow the dedicated Fernet and distributed-storage procedures.

Recovery

Preserve the database, all Fernet and credential key repositories, distributed storage, the configured KEK, and OPA policies. A database backup without its matching encryption material is not sufficient for recovery.

Command-line tools

Keystone-NG provides two command-line programs:

• keystone runs the API service or emits its OpenAPI schema.
• keystone-manage performs bootstrap, catalog, database, key, OAuth2, and distributed-storage administration.

Run either command with --help, and run nested keystone-manage commands with --help, for the exact options in the installed version.

keystone service command

keystone runs the service using /etc/keystone/keystone.conf by default.

keystone --config /etc/keystone/keystone.conf

The documentation workflow generates its schema with:

cargo run -p openstack-keystone --bin keystone -- --dump-openapi yaml \
  > doc/src/openapi.yaml

The generated file is ignored by Git and regenerated by the documentation workflow before each build.

keystone-manage administration command

keystone-manage performs administrative operations using the configured database, admin interface, key repositories, or distributed-storage endpoint.

keystone-manage --config /etc/keystone/keystone.conf <command>

Use <command> --help before execution. Follow the linked administrator runbook for Fernet tokens, OAuth2/OIDC, or distributed storage; command help describes syntax, while the runbook describes ordering and safety constraints.

Security Guidance

• Terminate public TLS at a trusted proxy and isolate the plain HTTP listener.
• Restrict the admin Unix socket and SPIFFE identities to trusted operators.
• Keep the metrics listener on a monitoring network.
• Keep OPA enabled in production and treat the policy bundle as trusted code.
• Store Fernet, credential, OAuth2 signing, audit, and distributed-storage keys as secrets; never print their contents during verification.
• Keep the audit spool directory ([audit] spool_dir) readable and writable by the Keystone user only, keep hmac-key.bin at mode 0600, and back it up like the Fernet keys. Investigate every *.quarantine-* file; Keystone never deletes one. See Audit trail.
• Keep insecure_allow_null_key = false except during a controlled migration.
• Use PKCS#11 or TPM KEK protection and authenticated transport for production distributed storage.
• Configure stable, cluster-wide WebAuthn decoy-key material to prevent account enumeration differences between nodes.
• Match identity mapping rules (SPIFFE, Kubernetes, federation, API client) on the narrowest claim that identifies the intended workload, and reserve is_system: true for genuine control-plane identities — an overbroad match or an unwarranted is_system grants real access, unlike an unmapped caller which is denied outright. See Identity Mapping Administration.

The contributor security model is normative for scope, delegation, rescope, credentials, tokens, and OPA input. Operator procedures must not weaken those invariants.

API Policy Enforcement

API policy is implemented using the Open Policy Agent (OPA). It is a very powerful tool and allows implementing policies much more complex than what the oslo.policy would ever allow. The policy folder contain default policies. They can be overloaded by the deployment.

OPA can be integrated into Keystone in 2 ways:

• HTTP. This is a default and recommended way of integrating applications with the OPA. Usually the OPA process is started as a side car container to keep network latencies as low as possible. Policies themselves are bundled into the container which OPA process is capable of downloading and even periodically refreshing. It can be started as opa run -s --log-level debug tools/opa-config.yaml. Alternatively the OPA process can itself run in the container in which case the configuration file should be mounted as a volume and referred from the entrypoint.

• WASM. Policies can be built into a WASM binary module. This method does not support feeding additional data and dynamic policy reload as of now. Unfortunately there is also a memory access violation error in the wasmtime crate happening for the big policy files. The investigation is in progress, so it is preferred not to rely on this method anyway. While running OPA as a WASM eliminates any networking communication, it heavily reduces feature set. In particular hot policy reload, decision logging, external calls done by the policies themselves are not possible by design. Using this way of policy enforcement requires wasm feature enabled.

All the policies currently are using the same policy names and definitions as the original Keystone to keep the deviation as less as possible. For the newly added APIs this is not anymore the case.

With the Open Policy Agent it is not only possible to define a decision (allowed or forbidden), but also to produce additional information describing i.e. reason of the request refusal. This is currently being used by the policies by defining an array of “violation” objects explaining missing permissions.

Sample policy for updating the federated IDP mapping:

package identity.mapping_update

# update mapping.

default allow := false

allow if {
	"admin" in input.credentials.roles
}

allow if {
	own_mapping
	"manager" in input.credentials.roles
}

own_mapping if {
	input.target.domain_id != null
	input.target.domain_id == input.credentials.domain_id
}

violation contains {"field": "domain_id", "msg": "updating mapping for other domain requires `admin` role."} if {
	identity.foreign_mapping
	not "admin" in input.credentials.roles
}

violation contains {"field": "role", "msg": "updating global mapping requires `admin` role."} if {
	identity.global_mapping
	not "admin" in input.credentials.roles
}

violation contains {"field": "role", "msg": "updating mapping requires `manager` role."} if {
	identity.own_mapping
	not "member" in input.credentials.roles
}

As can be guessed such policy would permit the API request when admin role is present in the current credentials roles or the mapping in scope is owned by the domain the user is currently scoped to with the manager role.`

List operation

All query parameters are passed into the policy engine to be provide capability of making decision based on the parameters passed. For example an admin user may specify domain_id parameter when the current authentication scope is not matching the given domain_id or a user with the manager role being able to list shared federated identity providers.

Policy is being evaluated before the real data is being fetched from the backend.

Show operation

Policy evaluation for GET operations on the resource are executed with the requested entity in the scope. This allows policy to deny the operation if the user requested resource it is should not have access to. This means that 404 error may be raised before the validation of whether the user is allowed to perform such operations.

Create operation

Resource creation operation would pass the whole object to be created in the context to the policy enforcement engine.

Update operation

For the update operation the context contain the current state of the resource and the new one. This allows defining policies preventing resource update upon certain conditions (i.e. when tag “locked” is added).

Delete operation

Resource deletion also passes the current resource state in the context to allow comprehensive logic.

Fernet Tokens

Fernet is the default token provider. Keystone-NG uses the same integer-indexed key repository model as Python Keystone so that both services can issue and validate tokens when they share an identical key set.

Fernet tokens are bearer credentials. Protect the key repository as secret material and use TLS for every client-facing connection.

Configuration

[token]
provider = fernet
expiration = 3600

[fernet_tokens]
key_repository = /etc/keystone/fernet-keys/
max_active_keys = 3
insecure_allow_null_key = false

expiration is the token lifetime in seconds. key_repository defaults to /etc/keystone/fernet-keys/, and max_active_keys defaults to 3.

insecure_allow_null_key is a migration-only escape hatch. Keep it false in production. Keystone refuses to start when the repository contains the well-known Null Key unless this option is explicitly enabled.

Key Repository

Each key is stored in a file named with an integer:

• 0 is the staged key for the next rotation.
• The highest positive index is the primary key used to encrypt new tokens.
• Lower positive indexes remain available to decrypt tokens issued before a rotation.

max_active_keys includes the staged key. With the default value of 3, the repository retains 0 and up to two positive-index keys. Choose the rotation schedule and retained-key count so a key is not pruned while tokens encrypted with it can still be valid.

All Rust and Python Keystone nodes must use the same complete file set. Do not initialize or rotate separate repositories independently on different nodes. Use a shared secret volume or distribute the complete repository to every node before a node begins issuing tokens with a new primary key.

Initial Setup

Run setup once against the deployment’s canonical repository before serving token requests:

keystone-manage --config /etc/keystone/keystone.conf token setup

The command creates key 0. It overwrites an existing staged key, so do not rerun it as a routine startup action. Run the command as the account that should own the repository; generated key files are written with owner-only permissions.

Verify file names, ownership, and modes without printing key contents:

find /etc/keystone/fernet-keys -maxdepth 1 -type f -printf '%f\n' | sort -n
stat -c '%a %U:%G %n' /etc/keystone/fernet-keys/*

Rotation

Rotate the canonical repository with:

keystone-manage --config /etc/keystone/keystone.conf token rotate

Rotation performs three operations:

1. Promotes staged key 0 to the next positive primary index.
2. Generates a new staged key at index 0.
3. Removes the oldest positive-index keys beyond max_active_keys.

Tokens are not re-encrypted. They remain usable only while the key that issued them is retained. The command does not check token lifetime before pruning, so operators must make the retention window longer than the maximum lifetime of tokens that may still be accepted.

The running Rust service watches the repository for changes and also polls it every 30 seconds. A valid rotation is loaded without restarting Keystone. If a reload fails, the service logs the failure and continues using the previous valid in-memory key set.

Multi-Node Rotation

For a shared repository, run the rotation command once. Every Rust process watching that repository reloads the resulting key set.

For repositories copied to each node:

1. Back up the current canonical repository securely.
2. Rotate one canonical copy.
3. Distribute the complete new file set to every Rust and Python node.
4. Confirm every node has the same indexes before returning normal traffic.

Do not rotate one node and leave it issuing tokens before the other nodes have the new primary key; those nodes will be unable to validate its tokens.

Troubleshooting

Keystone reports that no usable Fernet keys were found

• Confirm key_repository points to the mounted directory.
• Confirm setup has been run and the service account can read the key files.

Tokens fail validation after rotation

• Compare the key file indexes on every node.
• Check whether max_active_keys pruned a key before its tokens expired.
• Check the service log for a failed repository reload.

Keystone refuses to start because of the Null Key

• Replace the Null Key through a controlled repository migration.
• Do not enable insecure_allow_null_key as a production workaround.

Distributed Encrypted Storage

This guide covers the architecture, cryptographic design, and operational procedures for the Keystone-RS distributed storage engine. The design is specified in ADR 0016-v2 and implemented across four crates: openstack-keystone-distributed-storage (consensus, state machine, gRPC), openstack-keystone-storage-crypto (shared cryptographic primitives and the KekProvider trait), and the two production KEK providers, openstack-keystone-storage-crypto-pkcs11 and openstack-keystone-storage-crypto-tpm.

Table of Contents

1. Architecture Overview
2. Crate Layout
3. Key Hierarchy
4. Encryption Details
   • Raft Log Encryption
   • State Machine Encryption
   • Backup Encryption
   • Nonce Management
5. Data Tiers and Read Consistency
6. Intra-Cluster Transport (mTLS)
   • Peer Roles
7. Audit Log
8. Quarantine and GCM Failure Handling
9. DEK Rotation
10. Deployment Guide
    • Configuration Reference
    • PKCS#11 and TPM KEK Providers
    • First-Time Cluster Bootstrap
    • Adding Nodes
    • Kubernetes Reference Deployment
    • Single-voter k3s smoke test
    • TLS Certificate Management
11. Operational Runbook
    • Cluster Metrics
    • Scheduled DEK Rotation
    • Emergency DEK Rotation
    • Clearing a Quarantined Partition
    • Backup and Restore
12. Security Invariants

CLI Reference

All cluster management operations use keystone-manage storage <subcommand>. Every subcommand except init and join takes --cluster-addr <URI>, the cluster member to contact; it defaults to node_cluster_addr from the config file when omitted. init always targets that local address, and join takes the member to contact as its positional argument.

Leader targeting. Membership changes, DEK rotations, quarantine clearing, backup and restore into a running cluster must be proposed by the Raft leader. A follower answers these RPCs with UNAVAILABLE plus the leader’s id and address (x-openraft-leader-id, x-openraft-leader-endpoint), and the subcommands marked leader retry against that address, so --cluster-addr may name any member. The call is not proxied between nodes: the operator’s own identity reaches the leader, which is where the request is authorized, rate limited and audited. reconcile-dek-local-emergency (leader only) does not follow the redirect; it prints the leader’s address so the operator can first check that the leader holds the candidate. (*) Disaster-recovery restore into an uninitialized node runs on the given node.

demote and remove-peer of the current leader first hand leadership over to another voter (the same operation as transfer-leader), wait until it leads, and then apply the change on the new leader, so writes are not interrupted by an election. Run transfer-leader before a planned restart of the leader node for the same reason. A voter that lags the leader rejects the transfer; the command then tries the next voter. The transfer is audited as LEADERSHIP_TRANSFERRED. Successors are tried in node-id order, not ranked by replication lag. The leader cannot tell a target that refuses from one that simply does not answer, so each candidate that does not take over within 5 seconds (lagging, unreachable or not yet upgraded) adds that wait before the next one is tried; if none does, the error lists every candidate. If the transfer succeeds but the new leader cannot be reached afterwards, the command fails with a message saying so; the transfer is not rolled back, so retry the command.

transfer-leader relies on a node-to-node TransferLeader RPC added in this release. During a rolling upgrade a voter that is not yet upgraded answers it with UNIMPLEMENTED and never takes over. The transfer needs an upgraded leader (it serves the admin RPC) and at least one upgraded remaining voter; each non-upgraded voter tried before an upgraded one adds the wait above.

Architecture Overview

The storage engine combines three components:

All data is encrypted before it touches the Raft log or the Fjall on-disk storage. A full disk compromise or log exfiltration reveals only AES-256-GCM ciphertext — no plaintext, no key material.

┌─────────────────────────────────────────────────────────────────┐
│  Keystone API layer                                             │
└───────────────────────────────┬─────────────────────────────────┘
                                │  StorageApi trait
┌───────────────────────────────▼─────────────────────────────────┐
│  Storage struct (app.rs)                                        │
│  • Raft client  • DEK epoch  • Audit forwarder                  │
└───────┬───────────────────────┬─────────────────────────────────┘
        │ Raft proposals        │ Reads (ReadIndex)
┌───────▼───────────────────────▼─────────────────────────────────┐
│  OpenRaft (consensus)                                           │
│  ┌──────────────────┐         ┌────────────────────────────┐    │
│  │  FjallLogStore   │         │  FjallStateMachine         │    │
│  │  (log_store.rs)  │         │  (state_machine.rs)        │    │
│  │  Log DEK encrypt │         │  State DEK encrypt/decrypt │    │
│  └──────────────────┘         └────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────┘
        │                                       │
        ▼                                       ▼
  Fjall (log keyspace)                Fjall (state keyspace)
  [nonce][ciphertext][tag]            [nonce][ciphertext][tag][version]

Write Path

1. The API serializes the mutation to MessagePack.
2. The Log DEK encrypts the payload (nonce: node_id_BE ++ counter_BE, AD: term_BE ++ index_BE).
3. OpenRaft proposes the encrypted blob and replicates it over mTLS to a quorum.
4. On apply, the state machine decrypts the log entry using the Log DEK.
5. The current per-record version is read from Fjall (0 for new records).
6. The State DEK re-encrypts the value for at-rest storage (HKDF-derived nonce, AD: tier ++ domain ++ primary_key), then writes [nonce_12b][ciphertext][tag_16b][version_u32_BE] to Fjall.

Read Path

• Every tier: A ReadIndex (linearizable read) is issued to OpenRaft first, ensuring no stale follower can return a value that has since been revoked. Tiers are not carried on writes yet, so there is no cheaper local read path for Tier 0 / 1 (planned).

Crate Layout

crates/
├── storage-crypto/            # All cryptographic primitives
│   └── src/
│       ├── lib.rs             # Public re-exports
│       ├── kek.rs             # KekProvider trait, EnvKek (production
│       │                      #   providers: storage-crypto-pkcs11, -tpm)
│       ├── dek.rs             # DekEpoch, LogDek, StateDek, BackupDek, generate_dek
│       ├── cipher.rs          # log_encrypt/decrypt, state_encrypt/decrypt,
│       │                      #   backup_encrypt/decrypt
│       ├── nonce.rs           # NonceManager — durable monotonic counter
│       └── audit.rs           # AuditHmacKey
│
├── storage-crypto-pkcs11/     # Production KekProvider: PKCS#11 HSM/token
│   ├── src/lib.rs             # Pkcs11Kek, Pkcs11KekParams, SlotSelector
│   └── tests/softhsm.rs       # SoftHSM2-backed wrap/unwrap round-trip test
│
├── storage-crypto-tpm/        # Production KekProvider: TPM 2.0 resident key
│   ├── src/lib.rs             # TpmKek, TpmKekParams, KeyReference
│   └── examples/
│       └── tpm_kek_demo.rs    # Runnable sample against a software TPM (swtpm)
│
└── storage/                   # Consensus, gRPC, state machine
    └── src/
        ├── lib.rs             # StorageApi impl, DEK bootstrap
        ├── app.rs             # init_storage, Storage struct, StorageApi impl
        ├── preflight.rs       # OS-level memory protection checks
        ├── audit.rs           # AuditForwarder, AuditRecord
        ├── network.rs         # NetworkManager, SpiffeTlsProvider,
        │                      #   CertExpiryWatchdog, validate_svid_ttl
        ├── store/
        │   ├── log_store.rs   # FjallLogStore (OpenRaft LogStorage impl)
        │   └── state_machine.rs # FjallStateMachine (OpenRaft StateMachine impl)
        ├── grpc/
        │   ├── cluster_admin_service.rs  # init, add_learner, rotate_dek, …
        │   ├── raft_service.rs           # Raft RPC forwarding
        │   └── storage_service.rs        # Data read/write RPCs
        └── store_command.rs   # StoreCommand, MutationInner, DataTier

Key Hierarchy

 PKCS#11 HSM/token, or TPM 2.0 resident key  (production)
  │  or
  KEYSTONE_DEV_KEK env var  (dev mode only)
  │
  ▼
Key Encryption Key (KEK)                — never enters RAM as plaintext (prod)
  │
  │  AES-256-GCM unwrap
  ▼
Data Encryption Key (DEK)              — 256-bit random, mlock'd allocation
  │
  ├── Log DEK     HKDF(salt, DEK, "keystone-raft-log-v1",    L=32)
  ├── State DEK   HKDF(salt, DEK, "keystone-fjall-state-v1", L=32)
  └── Backup DEK  HKDF(salt, DEK, "keystone-backup-v1"
                               ++ dek_version_u32_BE,          L=32)

Audit HMAC key  HKDF(salt, DEK, "keystone-audit-dek-v1"
                              ++ dek_version_u32_BE
                              ++ node_id_u64_BE,               L=32)

The DEK sub-keys use full HKDF with the fixed salt keystone-dek-hierarchy-v1. The DEK is already uniformly random, so Extract adds no entropy; it is run because Extract-then-Expand is the flow the FIPS module documents as an approved service. Each sub-key is domain-separated by a distinct info string, ensuring ciphertexts from different contexts are never encrypted under the same key material.

The Audit HMAC key is derived from the DEK of the epoch it signs for, so it rotates with every DEK epoch swap, while remaining per-node to prevent cross-node forgery.

DEK Persistence

The wrapped DEK is stored in Fjall under the key _meta:dek:current as [version_u32_BE; 4] ++ wrapped_bytes. On startup, init_storage reads this key, unwraps the DEK under the KEK, derives sub-keys, and stores an Arc<RwLock<Arc<DekEpoch>>> that all state machine operations share.

Encryption Details

Raft Log Encryption

Function: log_encrypt(log_dek, plaintext, term, index) → Vec<u8>

On-disk layout: [nonce_12b][ciphertext][tag_16b]

The AD binding of term ++ index prevents an attacker from replaying a log entry from a different Raft position.

State Machine Encryption

Function: state_encrypt(state_dek, plaintext, tier, domain_id, pk, version) → Vec<u8>

On-disk layout: [nonce_12b][ciphertext][tag_16b][version_u32_BE]

The HKDF-derived nonce guarantees uniqueness across record updates: each (pk, version) pair produces a distinct nonce even if the same plaintext is re-written. The version starts at 0 for new records and increments on every write, stored as a 4-byte suffix alongside the ciphertext.

The tier byte in the AD cryptographically binds the sensitivity classification to the ciphertext — altering the stored tier makes the GCM tag invalid.

Write rate guard: If version >= write_rate_threshold (default 2^30, approximately 1 billion writes per record), further writes to that key are blocked with a WRITE_RATE_EXCEEDED violation and a CRITICAL log entry is emitted; a WARN is logged from 90% of the threshold. The per-record version keeps counting across DEK rotations (re-encryption increments it too), so a rotation does not lift the block for a record that reached it.

Backup Encryption

Function: backup_encrypt(bdek, snapshot_bytes, dek_version, utc_epoch, nonce_salt) → Vec<u8>

On-disk layout: [dek_version_u32_BE; 4] ++ [utc_epoch_u64_BE; 8] ++ [nonce_salt_u64_BE; 8] ++ [nonce_12b][ciphertext][tag_16b]

The dek_version and utc_epoch in the AD bind the snapshot to a specific point in time and DEK epoch, preventing time-travel and replay attacks across backup archives. nonce_salt is a fresh random 64-bit value generated per snapshot and mixed into the nonce derivation alongside utc_epoch, so two snapshots written within the same wall-clock second (including across a process restart, when a sequential in-memory counter would otherwise reset to 0) still get distinct nonces; it is stored directly in the header, so decryption reads it rather than searching for it. The header also carries a DEK manifest: the current DEK and every retired DEK the snapshot may still need, each wrapped under the KEK. A node holding the same KEK can therefore decrypt the backup even if it does not know these DEKs yet.

Nonce Management

The NonceManager (storage-crypto/src/nonce.rs) maintains a durable monotonic counter for Raft log nonces:

• Keeps one counter per DEK epoch, persisted node-locally in Fjall under _meta:nonce:<node_id>:ctr:<epoch> with its high-water mark under _meta:nonce:<node_id>:hwm:<epoch>. A DEK rotation therefore starts a fresh counter. A counter is never reset, so an epoch that becomes current again (after a restore) continues where it stopped. Snapshot installs and restores keep the node’s own counters.
• Nodes upgraded from the single per-node counter (_meta:nonce_ctr:<node_id>) record the epoch current at upgrade time; that epoch and every older one start no lower than the old counter.
• Reserves blocks of 1024 counts on each flush to absorb node crashes without nonce reuse.
• On startup, validates the recovered counter against the persisted high-water mark; refuses to start if the counter is behind the HWM (operator intervention required).
• Emits a WARN when fewer than 10% of the 2^31 threshold remain. From that point the leader rotates the DEK automatically (see DEK Rotation). At 2^31 the node can no longer append log entries under that DEK. The warning is rate limited, and the counter is exposed as keystone_raft_log_nonce_counter and keystone_raft_log_nonce_remaining.

Data Tiers and Read Consistency

Each record carries a DataTier marker (0–3) that is part of the AES-GCM associated data and stored in the record metadata.

Tier 2 and 3 always issue a ReadIndex RPC to the current Raft leader before reading from the local state machine, ensuring a revoked credential or removed group member can never be observed as still-valid on a lagging follower.

Local reads are off by default. Setting local_reads_mode = "local_for_public" lets a consumer ask for a stale local read of a single call; the node then serves Tier 0/1 records from its own state machine without contacting the leader and may return data older than the latest committed write; the staleness is unbounded, since a lagging or partitioned node serves whatever it last applied. Records without metadata or with an unknown tier are never served locally, and records written without an explicit tier default to Tier 1. Without the option, or for Tier 2/3 records (and prefix scans that match one), the request is ignored and the read goes through ReadIndex. Index key listings are always linearizable.

Intra-Cluster Transport (mTLS)

All cluster communication uses mutual TLS with the rustls defaults (TLS 1.2 is not disabled; the node does not restrict protocol versions or cipher suites itself). Every peer must present a valid mTLS identity, including a node that auto-joins through retry_join_nodes.

SPIFFE Mode (Default)

[distributed_storage]
trust_domains = "example.org"

• SVIDs issued by SPIRE are rotated automatically. TTL must not exceed 1 hour.
• Nodes reject SVIDs with less than 5 minutes remaining (force-renewal window).
• If SPIRE is unavailable before the renewal window, the node drains proposals and halts — it does not fall back to an expired SVID (fail-closed).
• Incoming SVIDs are mapped to a role (see Peer Roles); anything that maps to no role is rejected with PERMISSION_DENIED at the gRPC interceptor.
• Each node pins the SVID it presents: the first allowed_peer_svids entry’s path when that list is set (the cluster’s shared storage identity), else spiffe://<trust-domain><spiffe_path_prefix>node. Register that identity in SPIRE for every storage workload; if it is missing, storage initialization stalls in the SVID source retry loop and the node never becomes ready.
• allowed_peer_svids must be non-empty when dev_mode = false; the configuration is rejected otherwise.

TLS Fallback Mode

[distributed_storage]
tls_cert_file    = "/etc/keystone/storage/node.pem"
tls_key_file     = "/etc/keystone/storage/node.key"
tls_client_ca_file = "/etc/keystone/storage/ca.pem"

• Certificates must be signed by a dedicated Keystone Intermediate CA.

• Leaf certificate validity must not exceed 30 days.

• CertExpiryWatchdog checks remaining validity hourly: WARN at 7 days, ERROR at 2 days. It only logs; it does not shut the node down at expiry. It watches a certificate given inline (tls_cert_content), not one given as a file.

• The peer role is read from the first URI SAN of the client certificate. Set tls_role_san_prefix to the SAN URI prefix; the role name follows it. In TLS mode the role names are fixed to node and storage-operator; operator_role is not honoured:

  [distributed_storage]
  tls_role_san_prefix = "spiffe://keystone/storage/"
  # node cert SAN URI:     spiffe://keystone/storage/node
  # operator cert SAN URI: spiffe://keystone/storage/storage-operator
  

• Without tls_role_san_prefix the configuration is valid only with dev_mode = true, where every CA-signed peer is accepted for every RPC (legacy permissive behaviour, a warning is logged). With dev_mode = false it is a configuration error.

• The role is never derived from the certificate CN.

Peer Roles

Every inter-node and operator RPC is authorised by the role of the calling certificate. In SPIFFE mode (spiffe_path_prefix defaults to /keystone/storage/, operator_role to storage-operator):

The /ns/... SVID fallback has been removed. An allowed_peer_svids entry that lies under spiffe_path_prefix is resolved by the role convention, not by the allow-list, so it cannot be used to grant the operator role to an arbitrary name.

Which identity each command uses. In SPIFFE mode keystone-manage takes its SVID from the Workload API of the workload it runs in:

• join presents the node’s own SVID (allowed_peer_svids[0], else <spiffe_path_prefix>node) and therefore needs the Node role. Run it on the joining node, whose node_id and node_cluster_addr it announces.
• Every other subcommand presents the workload’s only SVID. list-peers and metrics accept Node or Operator; init, promote, demote, remove-peer and all other operator RPCs need Operator. Run these from an operator workload registered as <spiffe_path_prefix><operator_role> (and nothing else), not from a storage node: a node holds the Node role only, and a workload holding several SVIDs presents an arbitrary one. The workload needs a distributed_storage section so the CLI knows trust_domains and the cluster address (node_cluster_addr; subcommands with a --cluster-addr option can override it). The skaffold test cluster does this with the keystone-storage-operator Deployment (tools/k8s/tests/keystone-storage-operator.yaml).
• When SPIRE is run by the helm chart’s controller-manager, every ClusterSPIFFEID must set spec.className (default spire-spire); classless ones are silently ignored and the workload never receives the SVID.

The data-plane StorageService.Command RPC accepts only a Transaction made of Set, Remove, RemoveIndex, CreateIfAbsent and SetIndex mutations. Restore, DEK install, quarantine and rotation commands are rejected with PERMISSION_DENIED there; they are proposed in-process only.

Upgrade note (breaking change). Role resolution is stricter than in earlier builds. The role path convention is resolved first: any SVID under spiffe_path_prefix must be exactly <spiffe_path_prefix>node or <spiffe_path_prefix><operator_role>. An SVID such as spiffe://<td>/keystone/storage/node-1 is under the prefix with an unknown role name and is denied even when listed in allowed_peer_svids; re-register it as <spiffe_path_prefix>node. allowed_peer_svids is only for SVIDs outside the prefix (for example spiffe://<td>/ns/<namespace>/sa/<sa>), which must be listed verbatim. Before upgrading, register every storage workload under <spiffe_path_prefix>node, and the operator under <spiffe_path_prefix><operator_role>. In TLS mode, add the role SAN and set tls_role_san_prefix.

Rate limits

Per-identity rate limiters (for example 2 DEK rotations per hour, 10 ClearQuarantine calls per hour, 30 quarantine reports per hour) are held in memory on each node. They are not replicated through Raft and reset on restart. These RPCs are only served by the leader (a follower redirects before taking a token), so the limit applies per leader: it starts afresh after a leader change.

NodeId Uniqueness

Each node has a manually configured node_id: u64. At startup and on every add_learner gRPC call, the cluster membership is checked for a (node_id, rpc_addr) collision. A detected collision is fatal — the node or the operation is aborted with a clear error message. If membership cannot be queried (no quorum), the node logs a prominent warning and starts anyway: failing closed there would make recovery impossible when every node of the cluster restarts at the same time. The live check against a peer’s membership runs only when retry_join_nodes is configured.

Audit Log

Every security-relevant operation is signed and written to a durable, fsynced local spool. An external shipper (for example Filebeat or Vector) tails the spool and forwards it to the SIEM; the storage engine itself does not open a network connection to the SIEM.

Spool line structure (one JSON object per line):

{
  "record": {
    "timestamp": 1750000000,
    "event_type": "DEK_ROTATION",
    "actor": "operator@example.org",
    "node_id": 1,
    "dek_version": 3,
    "details": {}
  },
  "key_version": 3,
  "hmac": "<hex>"
}

Signature: HMAC-SHA256(AuditHmacKey, record_bytes), where record_bytes is the exact bytes of the record object as written in the line (do not re-serialise it before verifying).

Key derivation: the key is derived from the DEK of the epoch it signs for, not from the KEK: HKDF(salt, DEK, info = "keystone-audit-dek-v1" ‖ dek_version_u32_be ‖ node_id_u64_be). It rotates with every DEK epoch swap, on every node, and the key_version field names the epoch whose key signed the line — verifiers select the key by key_version. The node_id binding gives each node a distinct key. Because the key derives from the DEK, a verifier needs the epoch’s key material exported from a cluster node; export and retention of epoch keys is an operator responsibility and is not automated.

Availability: the spool is bounded by audit_max_spool_bytes (default 256 MiB, directory audit_spool_dir, default <path>/audit-spool). At 90% a CRITICAL alert is logged; at 100% the oldest sealed segment is deleted and an ERROR is logged. Spool contents are not encrypted at rest (files are created 0600). Spool exhaustion or write failure never blocks writes to the identity store, but dropped records are counted in keystone_raft_audit_dropped_total; the current spool size is exposed as keystone_raft_audit_spool_bytes.

Shipping: when [audit] sink is configured (see Configuration options), Keystone ships the sealed raft-audit-<node_id>.jsonl.seg-* segments to that sink verbatim and deletes each one after the sink acknowledged it; the live file is not touched. Delivery is at-least-once, so receivers should deduplicate. Without a sink, an external shipper must tail the spool as before. The records keep their own signature scheme described above; they are not CADF events.

Ordering and failures: a record is emitted only after the Raft write it describes has been attempted. A failed operation produces <EVENT>_FAILED with the error in details.error, never the success event. Every node additionally emits DEK_INSTALLED when it applies a DEK epoch swap.

Audited events include: DEK_ROTATION, DEK_ROTATION_EMERGENCY_STAGED, DEK_ROTATION_EMERGENCY_CONFIRMED, DEK_ROTATION_EMERGENCY_ABORTED, DEK_ROTATION_LOCAL_EMERGENCY_STAGED, DEK_ROTATION_LOCAL_EMERGENCY_RECONCILED, DEK_INSTALLED, LEADERSHIP_TRANSFERRED, CLUSTER_DEK_ADOPTED, QUARANTINE_CLEARED, BACKUP_CREATED, BACKUP_RESTORED, and the _FAILED variants of the Raft-committed operations.

Every restore attempt by an authenticated operator leaves exactly one record: BACKUP_RESTORED on success, BACKUP_RESTORED_FAILED when it is refused, aborted (stalled or disconnected upload, size overrun or mismatch), rejected by the cluster at apply time, or fails during a disaster recovery install. The failure record carries the mode (cluster or disaster_recovery, null if the attempt failed before it was known), declared_len, received_len, the gRPC code and the error message in error. It never contains backup contents or key material. Like the success record it is best effort and does not mask the restore error. A restore sent to a follower is redirected to the leader without a record: nothing was attempted on that node.

Quarantine and GCM Failure Handling

GCM tag verification failures indicate tampered or corrupted ciphertext.

Quarantine state is Raft-committed (stored in _meta:quarantine:<partition>:<node_id>) and therefore persists across restarts and is visible to all cluster members. A marker blocks reads only on the node that reported it; clear-quarantine clears the partition’s markers of all nodes. A restarted node reads this key at startup and re-enters quarantine if the marker is set.

Clearing quarantine requires a storage-operator identity:

keystone-manage storage clear-quarantine --partition <partition>

The clear operation is committed via Raft (so it takes effect cluster-wide) and is recorded in the audit log.

Reporting Quarantine from Followers

A follower that detects a GCM failure asks the leader to commit the quarantine marker through the dedicated StorageService.ReportQuarantine RPC (Node role). The leader validates the report against the current Raft membership, applies a rate limit of 30 reports per hour per reporter identity, and writes an audit record.

All nodes share the node role, so the reported node_id is not bound to the caller’s certificate. A compromised node can therefore report a quarantine for the partition of any cluster member. The exposure is bounded by the membership check, by the rate limit (which, because every node presents the same identity, is effectively one bucket per serving node), and by the audit record. Recovery is ClearQuarantine, which is operator-only. Binding each node to its own SVID is deferred (see ADR 0016-v2 addendum on peer roles).

DEK Rotation

The Raft leader rotates the DEK automatically when either holds:

• Age: the current DEK was installed dek_rotation_days ago (default 90; 0 disables this trigger). Each node records when it applied the DEK under _meta:dek:installed_at; nodes upgraded from a version that did not record it start the interval at their first start after the upgrade.
• Volume: the leader’s log nonce counter for the current DEK has used 90% of its 2^31 space. This trigger cannot be disabled.

The leader checks every 5 minutes and skips the check while an emergency rotation waits for its confirmation. An automatic rotation is the same InstallDek proposal as a manual one and is audited as DEK_ROTATION with the actor system:dek-rotation and the trigger in the details. A rotation proposes the next DEK version; when two rotations race for the same version (for example an automatic and a manual one), the second is rejected with a STALE_DEK_VERSION violation and the installed DEK is kept. The rotation is a live background process with no downtime.

Normal rotation:

keystone-manage storage rotate-dek

Emergency rotation (suspected DEK compromise — requires dual-control):

# Operator A initiates:
keystone-manage storage rotate-dek --emergency
# returns rotation_id=<uuid>

# Operator B confirms within 5 minutes:
keystone-manage storage confirm-rotate-dek --rotation-id <uuid>

If the 5-minute confirmation window expires without confirmation, the pending rotation is automatically aborted and an audit entry is written. Emergency rotations mark the old DEK as revoked (not retired) — it is never reused for any decryption, even for backup archives from that epoch.

Re-encryption: Each node re-encrypts its own Fjall records under the new DEK in a background task, keyspace by keyspace in key order. A record is read, re-encrypted with version + 1 and written while apply() is excluded from that record, so a concurrent Raft write is never reverted. A record that cannot be migrated after 3 attempts is skipped and logged at WARN. The sweep runs when a rotation is applied and again every 5 minutes for every retired DEK that still has records under it, so skipped records are retried without waiting for the next rotation. A pass that finds nothing left to migrate marks the retired DEK as done (_meta:dek:reencrypt_done:<version>). Retired DEKs are kept for reading older snapshots and backups.

Progress: Every 1000 records the sweep checkpoints its position and counts to _meta:dek:rotation_progress:<node_id>:<version>. After a restart the next pass resumes behind the checkpoint, and records skipped before the restart still keep the retired DEK from being marked done. The checkpoint is removed when a pass completes and is not carried by snapshots.

Not implemented: a post-rotation verification report, the CRITICAL alert for records left unmigrated for 24 hours, and blocking DEK retirement until they are resolved. Skipped records are only visible in the WARN logs and are retried by the periodic sweep.

Deployment Guide

Prerequisites

• Rust toolchain (see rust-toolchain.toml)
• A SPIRE deployment, or TLS certificates from a dedicated Intermediate CA
• For production: a PKCS#11 HSM/token (kek_provider = "pkcs11") or a TPM 2.0 chip (kek_provider = "tpm") for KEK storage — see PKCS#11 and TPM KEK Providers
• For development: set KEYSTONE_DEV_KEK and KEYSTONE_ALLOW_ENV_KEK=1

Configuration Reference

[distributed_storage]
# Unique identifier for this node within the cluster. Must be a u64.
# Collision with an existing node at a different address is fatal.
node_id = 1

# Advertised cluster-internal address (used by peers for Raft RPC).
node_cluster_addr = "https://10.0.0.1:8310"

# Local listener address for inbound cluster connections
# (default: 0.0.0.0:8081).
node_listener_addr = "0.0.0.0:8310"

# Development mode: accepts the env KEK and, in TLS mode without
# tls_role_san_prefix, any CA-signed peer. Must be false (default) in
# production; tools/check_no_dev_mode.sh flags service definitions that set it.
# dev_mode = false

# Peers to auto-join on startup, as comma separated `id=address` pairs. Node
# id 0 is the bootstrap node: it initializes itself with the full list as the
# initial membership. Any other node adds itself as a learner on the first
# listed peer that answers (after adopting the cluster DEK). The join is still
# authenticated by mTLS. Without the list the nodes are joined manually with
# `init` and `join`. All nodes must share the same KEK material.
# retry_join_nodes = "0=https://keystone-0:8310,1=https://keystone-1:8310,2=https://keystone-2:8310"

# Set to false on nodes that are about to receive a disaster recovery
# `restore`, so that they are not initialized on first start (default: true).
# auto_bootstrap = true

# Reads that cannot be confirmed linearizable are retried this many times
# (default: 80) with this delay in milliseconds (default: 50) before they
# fail with `Unavailable`.
# ensure_linearizable_retries = 80
# ensure_linearizable_retry_delay_ms = 50

# Allow consumers to request stale local reads of Tier 0/1 records
# (`disabled` (default) or `local_for_public`).
# local_reads_mode = "disabled"

# Raft timing and log compaction. Raise the heartbeat and election timeouts for
# clusters spread across availability zones. `raft_snapshot_logs_since_last = 0`
# disables automatic snapshots. Defaults shown.
# raft_heartbeat_interval_ms = 500
# raft_election_timeout_min_ms = 1500
# raft_election_timeout_max_ms = 3000
# raft_snapshot_logs_since_last = 5000
# raft_max_in_snapshot_log_to_keep = 1000
# raft_purge_batch_size = 1

# Audit spool directory (default: <path>/audit-spool) and its size bound
# (default: 256 MiB).
# audit_spool_dir = "/var/lib/keystone/storage/audit-spool"
# audit_max_spool_bytes = 268435456

# Directory where Fjall database files are stored.
path = "/var/lib/keystone/storage"

# Age in days after which the leader rotates the DEK automatically
# (default: 90). 0 disables the age trigger; the log nonce volume trigger
# always stays on.
dek_rotation_days = 90

# Per-record write version at which further writes to the record are
# rejected (default: 2^30).
write_rate_threshold = 1073741824

# Selects the production KEK source. "env" (default) is dev-mode only and is
# rejected unless dev_mode = true. See "PKCS#11 and TPM KEK Providers" below.
# kek_provider = "pkcs11"
# kek_provider = "tpm"

# --- Transport: SPIFFE (default) ---
trust_domains = "example.org"
# SVIDs outside spiffe_path_prefix that are accepted with the node role.
# Required when dev_mode = false. The first entry is also the SVID this node
# itself presents.
# allowed_peer_svids = "spiffe://example.org/keystone/storage/node"
# Role of the operator workload (default: storage-operator) and the path prefix
# of all storage SVIDs (default: /keystone/storage/).
# operator_role = "storage-operator"
# spiffe_path_prefix = "/keystone/storage/"

# --- Transport: TLS fallback ---
# tls_cert_file    = "/etc/keystone/storage/node.pem"
# tls_key_file     = "/etc/keystone/storage/node.key"
# tls_client_ca_file = "/etc/keystone/storage/ca.pem"
# # Or embed content directly (base64 or PEM):
# tls_cert_content = "..."
# tls_key_content  = "..."
# tls_client_ca_content = "..."
# URI prefix of the SAN that carries the peer role (required when
# dev_mode = false), e.g. spiffe://keystone/storage/
# tls_role_san_prefix = "spiffe://keystone/storage/"

# The [distributed_storage.pkcs11] and [distributed_storage.tpm] subsections
# are described below; node-local emergency writes (quorum loss) are configured
# in the separate [local_emergency] section (ADR 0028).

Environment variables (development only):

Warning: KEYSTONE_DEV_KEK and KEYSTONE_ALLOW_ENV_KEK must never appear in production Dockerfiles, Kubernetes manifests, or systemd units. tools/check_no_dev_mode.sh checks for this and runs in the Linters CI workflow.

PKCS#11 and TPM KEK Providers

Production deployments select one of the two hardware-backed KekProvider implementations (ADR 0016-v2 §2.5). Both wrap/unwrap the DEK with CKM_AES_GCM/TPM2 AES-GCM directly against a non-extractable AES-256 key object — the key material never leaves the token or chip.

PKCS#11 (HSM or token)

[distributed_storage]
kek_provider = "pkcs11"

[distributed_storage.pkcs11]
# Path to the vendor's (or SoftHSM2's) Cryptoki shared library.
pkcs11_module_path = "/usr/lib/softhsm/libsofthsm2.so"

# CKA_LABEL of the AES-256 key object. Must have CKA_EXTRACTABLE = false
# (ADR 0016-v2 §10 invariant 13). init_storage never creates this key —
# it must already exist via an operator's out-of-band provisioning step.
pkcs11_key_label = "keystone-kek"

# Either the slot id or the token label must be given; label is preferred
# since slot ids can shift across token re-initialisation.
pkcs11_slot_label = "keystone-storage"
# pkcs11_slot_id = 0

# File containing the token PIN. Never accepted inline or via env var
# (ADR 0016-v2 §10 invariant 14).
pkcs11_pin_file = "/etc/keystone/storage/pkcs11.pin"

# Let the token generate the GCM IV when wrapping the DEK (PKCS#11 3.0
# message API, CKG_GENERATE_RANDOM). Needed for FIPS 140-3 HSMs that reject
# caller-supplied IVs. The token must implement C_MessageEncryptInit
# (SoftHSM2 does not). Default: false.
# pkcs11_token_generated_iv = true

Provisioning the key (a one-time operator ceremony, run once per token before the cluster’s first boot) uses any Cryptoki client capable of generating a non-extractable AES-256 key under the target label — for example, pkcs11-tool --keygen --key-type AES:32 --label keystone-kek, or the provisioning helper in crates/storage-crypto-pkcs11/tests/softhsm.rs and crates/storage/tests/test_pkcs11_cluster.rs, which do the equivalent programmatically against SoftHSM2 for local testing.

Local testing against SoftHSM2:

# libsofthsm2.so and the softhsm2-util CLI:
sudo apt-get install -y softhsm2

# Run the SoftHSM2-backed tests (skip themselves if the module isn't found):
cargo test -p openstack-keystone-storage-crypto-pkcs11
cargo test -p openstack-keystone-distributed-storage --features pkcs11 --test test_pkcs11_cluster

TPM 2.0

[distributed_storage]
kek_provider = "tpm"

[distributed_storage.tpm]
# TCTI connection string: a hardware TPM's resource manager device, or a
# software TPM (swtpm) for testing.
tpm_tcti = "device:/dev/tpmrm0"

# Exactly one of the following identifies the pre-provisioned AES-256 key:
tpm_key_handle = "0x81000001"      # persistent handle, or:
# tpm_key_context_file = "/etc/keystone/storage/tpm-kek.ctx"

# Optional: file containing the key's auth value (userWithAuth keys only).
# tpm_auth_file = "/etc/keystone/storage/tpm.auth"

As with PKCS#11, init_storage only ever opens the key with auto_generate: false — provisioning is a separate operator step. crates/storage-crypto-tpm/examples/tpm_kek_demo.rs is a runnable sample that provisions and exercises a KEK against a software TPM:

# 1. Start a software TPM:
mkdir -p /tmp/swtpm-state
swtpm socket --tpmstate dir=/tmp/swtpm-state \
    --ctrl type=tcp,port=2322 --server type=tcp,port=2321 \
    --tpm2 --flags not-need-init &
TPM2TOOLS_TCTI="swtpm:host=127.0.0.1,port=2321" tpm2_startup -c

# 2. Run the sample (first run provisions the key, later runs reload it):
cargo run -p openstack-keystone-storage-crypto-tpm --example tpm_kek_demo

This example is compiled in CI on every run to catch rot, but is not executed there — real/virtual TPM availability isn’t reliable enough on shared CI runners to gate merges on (ADR 0016-v2 §2.5.2).

First-Time Cluster Bootstrap

Step 1 — Start each node (do not initialize yet):

keystone --config /etc/keystone/keystone.conf

Each node starts and waits; Raft is not yet initialized.

Step 2 — Initialize the first node as a single-node cluster.

init registers the node_id and node_cluster_addr of the config it runs with as the first member, so its config must describe node 1. In SPIFFE mode it also needs the Operator role (see CLI Reference): run it from an operator workload whose distributed_storage section carries node 1’s values:

keystone-manage storage init

Node 1 becomes the leader of a 1-node cluster. Wait for it to report a leader (check keystone-manage storage metrics --cluster-addr https://10.0.0.1:8310).

Step 3 — Add learners.

Run from node 2 and node 3’s hosts respectively. The positional argument is the address of any existing cluster member to contact:

# On node 2's host:
keystone-manage storage join https://10.0.0.1:8310

# On node 3's host:
keystone-manage storage join https://10.0.0.1:8310

join first asks the local node, which must already be running, to fetch the cluster’s data encryption keys from the leader and install them, and only then registers it as a learner. The node therefore never replicates under the DEK it generated at first boot. All nodes must share the same KEK material, otherwise the adoption fails and the join is aborted. Re-running join on a node that is already a member is harmless. If the registration fails after the adoption (for example because the id is registered at a different address), the node keeps the adopted DEK, which is correct for the cluster; fix the cause and run join again. The local node must run a version that supports the adoption, otherwise join stops with an error asking you to upgrade it first.

Step 4 — Promote learners to voting members.

Run from an operator workload (SPIFFE mode: storage-operator SVID). The command locates the leader through --cluster-addr (default: the workload’s node_cluster_addr), so it can contact any member. Repeat once per learner to promote:

keystone-manage storage promote 2
keystone-manage storage promote 3

Adding Nodes

To add a new node to a running cluster:

# 1. Start the new node process (it will wait for a join instruction).

# 2. On the new node's host, join to an existing cluster member:
keystone-manage storage join https://10.0.0.1:8310

# 3. Optionally promote to voting member (run from an operator workload):
keystone-manage storage promote 4

Kubernetes Reference Deployment

tools/k8s/keystone/overlays/production is the reference for running the Raft storage on Kubernetes. The skaffold overlay is for development only: it runs in dev mode with an environment KEK and keeps the storage on an emptyDir.

kubectl apply -k tools/k8s/keystone/overlays/production/spire
kubectl apply -k tools/k8s/keystone/overlays/production

What it sets up, and why:

• Persistent storage. path (/storage) is a volumeClaimTemplates volume. It holds the Raft log, the state machine, the wrapped DEK, the nonce high-water mark (_meta:nonce_hwm), the quarantine markers and the audit spool. On an emptyDir a restarted pod comes back as a new node with the same node_id and a new DEK, and the nonce rollback protection is void.
• Three voters. replicas: 3 and a matching retry_join_nodes list. Keep both in sync and use an odd number. podManagementPolicy: Parallel starts all members together; rolling updates still replace one pod at a time and wait for it to become ready.
• Disruption and placement. A PodDisruptionBudget with maxUnavailable: 1 keeps a quorum through node drains, required podAntiAffinity keeps two members off the same node, and a topologySpreadConstraints entry spreads them over zones.
• Identities. The StatefulSet runs under its own keystone-rs service account. The ClusterSPIFFEID entries in spire/ bind the <spiffe_path_prefix>node identity to that namespace, label and service account, so no other workload can become a Raft peer. Operator commands run from the keystone-storage-operator Deployment, the only workload holding <spiffe_path_prefix>storage-operator. It ships with replicas: 0; scale it up for a maintenance window only.
• Trust domain. The SPIRE registrations in spire/ use {{ .TrustDomain }} and adapt to your cluster, but the keystone-side config hardcodes the example.org placeholder. Before applying, replace every example.org with your SPIRE trust domain in: the init-container heredoc in statefulset-patch.yaml (trust_domains, allowed_peer_svids), conf/keystone.conf (interface_internal.trust_domains, interface_admin.trust_domains, interface_admin.admin_svid), and storage-operator.yaml (trust_domains, allowed_peer_svids). A mismatch silently breaks intra-cluster mTLS and admin auth, because SVIDs that do not match the pinned values are rejected.
• KEK. kek_provider = "pkcs11" with the token PIN mounted from the keystone-pkcs11-pin Secret (create it out of band). The published ghcr.io/openstack-experimental/keystone:main image is built with both the PKCS#11 and TPM features (see .github/workflows/container-publish.yml), so this overlay works as-is and a TPM-backed cluster can switch the config to kek_provider = "tpm" without changing the image. A self-built image must pass the same build argument, e.g. docker build --build-arg KEYSTONE_FEATURES=openstack-keystone/pkcs11,openstack-keystone/tpm . (add openstack-keystone/wasm_plugins as well if the image should keep dynamic auth plugin support, as the published image does). The image must also contain the vendor’s PKCS#11 module and client configuration at pkcs11_module_path. For a TPM, set kek_provider = "tpm" and give the pod access to the TPM device (/dev/tpmrm0), typically through a device plugin or, for a single-node test, a hostPath char-device mount; the image already ships the libtss2 runtime libraries required by tss-esapi.
• Pre-flight. The pods run as non-root with all capabilities dropped and a read-only root file system. Keystone sets RLIMIT_CORE = 0 and PR_SET_DUMPABLE = 0 itself; neither needs a privilege. The container command raises the RLIMIT_MEMLOCK soft limit to the hard limit set by the container runtime. Outside of dev mode the node refuses to start when that is below 64 KiB; raise the runtime’s default (e.g. LimitMEMLOCK of the containerd systemd unit, default_ulimits in CRI-O) if it is. CAP_IPC_LOCK is not needed and has no effect for a non-root process.

After the first rollout keystone-rs-0 (node_id = 0) has initialized the cluster and the other members have joined as learners. Promote them from the operator workload:

kubectl -n keystone scale deploy/keystone-storage-operator --replicas=1
kubectl -n keystone exec deploy/keystone-storage-operator -- keystone-manage storage promote 1
kubectl -n keystone exec deploy/keystone-storage-operator -- keystone-manage storage promote 2
kubectl -n keystone scale deploy/keystone-storage-operator --replicas=0

keystone-rs-0 stays the bootstrap node: whenever it starts with an empty volume it initializes a new single-node cluster, splitting it off from the running one. If its volume is lost, start it with auto_bootstrap = false so it stays uninitialized, remove the stale member with keystone-manage storage remove-peer 0 from the operator workload, run keystone-manage storage join <address of a live member> in keystone-rs-0 and promote it again (see Adding Nodes).

The readiness probe uses /ready, which returns 503 while the storage is not initialized or reports a recoverable Raft issue (no known leader, a stale quorum acknowledgement, or a quarantined partition). The liveness probe uses /health, which fails only on an unrecoverable error such as a stopped Raft core.

Single-voter k3s smoke test

tools/k8s/keystone/overlays/k3s-test is a throwaway, single-voter overlay for verifying the two production KEK providers on a single-node k3s cluster. It is not a quorum test: the overlay drops the required pod anti-affinity and runs only keystone-rs-0.

The overlays render the same dependencies as the other Keystone overlays: a CloudNativePG Cluster for the bootstrap job’s database, SPIRE ClusterSPIFFEID registrations, and the keystone-py image used by the bootstrap job. Install CloudNativePG and the SPIRE controller manager on the k3s cluster first (the main keystone Skaffold module installs both).

Use the Skaffold modules described in the Raft KEK overlays on k3s section of the contributor development guide:

skaffold run -f skaffold-k3s-test.yaml \
  -p pkcs11 --default-repo localhost:5000 --tag raft-test --cleanup=false
skaffold run -f skaffold-k3s-test.yaml \
  -p tpm --default-repo localhost:5000 --tag raft-test --cleanup=false

The PKCS#11 variant uses SoftHSM2. It initialises a token on the StatefulSet volume and generates the keystone-kek AES-256 key automatically.

The TPM variant runs a swtpm sidecar on pod loopback and sends TPM2_Startup before starting keystone. This avoids depending on a host TPM, which may lack TPM2_EncryptDecrypt2 (for example, some ARM/virt implementations return 0x000b0143 — “command code not supported” — for it). Provision the sidecar state and the KEK context files on the k3s host before deploying the TPM module; the Skaffold module creates the keystone-tpm-kek Secret from those files.

A real host TPM can be used instead by pointing tpm_tcti at device:/dev/tpmrm0, removing the swtpm sidecar and the tpm2_startup call from tools/k8s/keystone/overlays/k3s-test/tpm/statefulset-patch.yaml, and provisioning the context files against the host TPM. The TPM must support TPM2_EncryptDecrypt2.

Tear down either variant with:

skaffold delete -f skaffold-k3s-test.yaml \
  -p pkcs11 --default-repo localhost:5000
skaffold delete -f skaffold-k3s-test.yaml \
  -p tpm --default-repo localhost:5000

The StatefulSet volume claim is deleted with the namespace.

TLS Certificate Management

SPIFFE mode: No operator action required. SPIRE rotates SVIDs automatically. The node refuses connections from SVIDs with < 5 minutes remaining validity.

TLS fallback mode:

1. Generate a new certificate from your Intermediate CA (max 30-day validity).
2. Deploy the new certificate and key to the node.
3. Restart the node, or use a runtime reload mechanism if available.
4. The CertExpiryWatchdog logs WARN at 7 days remaining and ERROR at 2 days.

Operational Runbook

Cluster Metrics

Quick health check — shows current leader, voter set, and raw OpenRaft metrics:

keystone-manage storage metrics --cluster-addr https://10.0.0.1:8310

Sample output (the raw metrics are one long line, shown as printed by the contacted node):

Current leader : node 1
Voters         : [1, 2, 3]
All nodes      : [1=10.0.0.1:8310, 2=10.0.0.2:8310, 3=10.0.0.3:8310]

Raw metrics:
Metrics{id:1, Leader, term:1, vote:<T1-N1:Q>, last_log:5, local_committed:1.5, cluster_committed:1.5, last_applied:1.5, leader:1(quorum_acked_time:10:32:00.294800, 483.80772ms ago), membership:{log_id:1.5, {voters:[{1:Node { node_id: 1, rpc_addr: "10.0.0.1:8310" },2:Node { node_id: 2, rpc_addr: "10.0.0.2:8310" },3:Node { node_id: 3, rpc_addr: "10.0.0.3:8310" }}], learners:[]}}, committed_membership:{log_id:1.5, {voters:[{1:Node { node_id: 1, rpc_addr: "10.0.0.1:8310" },2:Node { node_id: 2, rpc_addr: "10.0.0.2:8310" },3:Node { node_id: 3, rpc_addr: "10.0.0.3:8310" }}], learners:[]}}, snapshot:None, purged:None, replication:{1:1.5,2:1.5,3:1.5}, heartbeat:{1:10:32:00.294800,2:10:32:00.294800,3:10:32:00.294800}}

For a formatted peer table use list-peers instead.

Storage Status

The DEK, rotation, quarantine and nonce state the runbooks below refer to is reported per node by status:

keystone-manage storage status --cluster-addr https://10.0.0.2:8310

Sample output:

Node                  : 2
State                 : Follower
Current leader        : 1
Current term          : 3
Last log index        : 1042
Last applied index    : 1042
DEK version           : 4
Retired DEK versions  : 2, 3
Revoked DEK versions  : none
Pending rotations     : none
Quarantined (local)   : none
Quarantine records    : none
Nonce counter         : 18432 / 2147483648 (0.00%)

Quarantined (local) lists the partitions whose reads are blocked on this node; Quarantine records lists every marker the node holds, including those reported by other nodes. Pending rotations shows emergency rotations awaiting confirm-rotate-dek (authoritative on the leader). The nonce counter is the persisted reservation point (at most 1024 ahead of use); the node stops accepting log writes when it reaches the threshold. Run the command against each node of interest: DEK state can briefly differ while a follower applies a rotation, and quarantine is node-local.

Scheduled DEK Rotation

Automatic rotation fires after dek_rotation_days (default: 90) or when the leader’s log nonce counter for the current DEK reaches 90% of 2^31 (see DEK Rotation). Manual rotation:

keystone-manage storage rotate-dek \
  --cluster-addr https://10.0.0.1:8310

Monitor the audit log (event_type = "DEK_ROTATION"; automatic rotations use the actor system:dek-rotation) and the node logs for records the re-encryption sweep skipped (record skipped after exhausting CAS retries).

Emergency DEK Rotation

Use when a DEK is suspected compromised.

# Operator A — initiates rotation, receives rotation_id:
keystone-manage storage rotate-dek \
  --cluster-addr https://10.0.0.1:8310 \
  --emergency
# Output: rotation_id=550e8400-e29b-41d4-a716-446655440000

# Operator B — confirms within 5 minutes:
keystone-manage storage confirm-rotate-dek \
  --cluster-addr https://10.0.0.1:8310 \
  --rotation-id 550e8400-e29b-41d4-a716-446655440000

If no confirmation is received within 5 minutes, the rotation aborts automatically and is recorded in the audit log. The dek_rotation_days timer restarts when the confirmed rotation installs the new DEK.

Clearing a Quarantined Partition

A partition enters quarantine after 3 GCM verification failures within 60 seconds. In quarantine, the partition is read-only and all writes to affected keys are rejected with a QUARANTINED violation.

Diagnosis:

# Check the quarantine state of the affected node:
keystone-manage storage status --cluster-addr https://10.0.0.1:8310

Root-cause the GCM failures (hardware fault, storage corruption, or unauthorized modification) before clearing quarantine.

Clear:

keystone-manage storage clear-quarantine \
  --cluster-addr https://10.0.0.1:8310 \
  --partition <partition-name>

Note: the partition is now the --partition option (default data). Scripts that passed it positionally (clear-quarantine <partition>) must be updated.

This commits a Raft proposal (visible cluster-wide) and emits an audit entry.

Backup and Restore

Create a backup (Fjall snapshot):

keystone-manage storage backup \
  --cluster-addr https://10.0.0.1:8310 \
  --output /mnt/backups/keystone-$(date +%Y%m%d).snap

The command triggers a fresh Fjall snapshot on the target node, then streams the AES-256-GCM encrypted bytes to --output. The final output includes the snapshot_utc_epoch and dek_version printed on completion for verification.

The snapshot is wrapped in a backup-specific AES-256-GCM envelope with the Backup DEK, bound to the snapshot timestamp and current DEK epoch. Its header carries the DEK manifest: the current and retired DEKs, wrapped under the KEK.

Restore into a running cluster (the usual case):