Audit trail

Keystone writes a signed, tamper-evident audit record for every authentication attempt and every state-changing operation. This page is the operator guide; the design is in ADR 0023.

What is audited

Ordinary validated-token requests to resource endpoints are not recorded at the perimeter; the mutations they make are audited by the provider records.

A few bookkeeping operations are deliberately not audited because the authentication that triggers them is: the last-use timestamp and transparent secret re-hash of an API key, and janitor housekeeping.

Record format

Records are JSON lines using the DSP0262 (CADF) key names:

{
  "typeURI": "http://schemas.dmtf.org/cloud/audit/1.0/event",
  "eventType": "activity",
  "id": "keystone-0:550e8400-e29b-41d4-a716-446655440000",
  "eventTime": "2026-06-16T10:15:00+00:00",
  "action": "delete",
  "outcome": "failure",
  "reason": {"reasonType": "keystone", "reasonCode": "Conflict"},
  "initiator": {
    "typeURI": "service/security/account/user",
    "domain_id": "0123456789abcdef0123456789abcdef",
    "host": {"address": "203.0.113.9"},
    "id": "fedcba9876543210fedcba9876543210",
    "project_id": null
  },
  "target": {"id": "0123456789abcdef0123456789abcdef", "typeURI": "data/security/identity/user"},
  "observer": {"id": "service/security/keystone/keystone-0", "typeURI": "service/security/keystone"},
  "tags": ["correlation_id:req-3f0c7a3e5b3d4d0f9a52f1c8f1f1e0aa"],
  "attachments": [{
    "name": "integrity",
    "contentType": "application/json",
    "content": {
      "seq": 42,
      "boot_session_id": "9a1c3f0e-6c0b-4c5d-8f55-2f2c3b6d2a10",
      "hmac_key_version": 1,
      "version": "1.1",
      "domain": "0123456789abcdef0123456789abcdef",
      "observer_node_id": "keystone-0"
    }
  }],
  "signature": "<hex HMAC-SHA256>"
}

• outcome is one of the DSP0262 values success, failure, pending (provider record written before the change) or unknown; no other value is written, and a record carrying one is rejected on read. A rejected request such as a rate limit is a failure whose reason.reasonCode is TooManyRequests or ClientError. Rate-limited or otherwise denied plugin invocations are likewise a failure with the cause in reason.reasonCode.
• reason.reasonCode is a fixed vocabulary word (an error variant name such as Conflict or UserLocked), never error text. The reason key is omitted when there is none.
• OAuth2 token-endpoint records (authenticate for the authorization_code, refresh_token and device_code grants) carry a second attachments entry, oauth2, with content {"client_id": "...", "grant_type": "..."}. It is covered by the signature. Look attachments up by name, never by position. A rejected device_code client is a failure whose reason.reasonCode is the OAuth2 error (invalid_client, unauthorized_client).
• action is a lowercase verb or a /-separated name such as oauth2/refresh_reuse_detected.
• initiator.host carries the client address and, for pre-authentication failures, a non-secret identifier such as the EC2 access key id. It is omitted when empty, whereas project_id and domain_id are null.
• The correlation id is the correlation_id: tag: the server-generated x-openstack-request-id; a client-supplied value is discarded.
• The integrity attachment carries the sequence number, boot session, key version, schema version, the initiator’s domain and the observer node.
• Records contain identifiers only: no passwords, tokens, secrets or free text.

The spool

Records are first written to a local spool in [audit] spool_dir (default /var/lib/keystone/audit), so nothing depends on the downstream system being up.

The spool grows until segments are shipped. Configure a sink so sealed segments are shipped and deleted once the sink has accepted them; without one, segments stay in the spool for an external shipper (for example a log forwarder tailing the directory) and nothing deletes them unless you set spool_max_segments (the oldest are then deleted with an ERROR log). Plan disk for spool_max_segments * spool_max_segment_bytes.

Configuration

Keys

hmac-key.bin holds at least 32 random bytes. It is created with mode 0600 on first start; a file that is too short makes startup fail. Each node derives its own signing key from it with HKDF-SHA256 (Extract-then-Expand) using the salt keystone-audit-hmac-salt-v1 and the info string keystone-audit-hmac-v1:<node_id>, so a compromised node cannot forge records attributed to another node. Back the file up with the same care as the Fernet keys: without it the spool cannot be verified, with it an attacker can forge records. The key version in use is exported as keystone_audit_hmac_key_version and stamped on every record (hmac_key_version in the integrity attachment), so old records stay verifiable; automated rotation is not available yet.

SIEM implementors reproduce the key as HKDF-SHA256(salt, IKM = KEK, info, L = 32) with the standard RFC 5869 Extract and Expand steps. Releases before this derivation used Expand-only HKDF (no salt); their records do not verify with the key above.

FIPS builds

Build with the cadf/fips feature (--no-default-features --features fips to leave ring out) to run the HMAC signature, the key derivation and KEK generation on the FIPS 140-3 validated AWS-LC module. Call cadf::assert_fips() at start-up to refuse to run without it. The syslog TLS stack follows the service’s fips feature (the TLS provider crate), not this one.

Verifying records in a SIEM

1. Parse the JSON line.
2. Remove the signature key.
3. Serialize the rest in JCS canonical form (RFC 8785).
4. Compute HMAC-SHA256 with the key for hmac_key_version (and node_id).
5. Compare with signature.

Cache every key version you have seen. Reference vectors, including the host cases, are in tests/audit/hmac_vectors.jsonl, and tools/audit_vectors.py verify is an independent Python implementation of the procedure. Cross-check with outcome: a pending record with no success/failure for the same correlation id and target within a few minutes means the process died or the post-record was lost.

Monitoring

Keystone exports keystone_audit_* metrics on /metrics (event, drop, queue-depth, spool and sink series; see ADR 0023 §Observability). Ready-made alert rules are in deploy/prometheus/alert_rules.yaml; at minimum page on keystone_audit_dropped_total, keystone_audit_postaudit_dropped_total, keystone_audit_spool_write_failures_total, keystone_audit_spool_quarantined_total and sink errors.

Accepted risk: records live in memory for a short window before they reach the spool, and the spool is not fsynced per record, so a hard crash or power loss can lose the last few milliseconds of records. A graceful shutdown drains the queue first.