Key custody
Trace who holds which keys where: the two KMS trust domains, how each plane generates and stores its keys, what keeps the controlplane out of enclave key custody, and what rotation and destruction look like.
Daevix encrypts data at rest with envelope encryption, and the security of that encryption rests on one question: who can reach the keys? This page answers it plane by plane. The threat model covers the same ground adversarially, boundary by boundary; this page states the same facts plainly.
The governing rule
Key custody never crosses a boundary that the data itself may not cross. The
controlplane never touches sovereign data, so the controlplane never touches
the keys that guard it. The repository’s docs/architecture.md states this as an
invariant, and threat model section 4.6 analyzes it adversarially. This page
summarizes both without replacing them.
Two KMS trust domains, never crossing
Each plane owns exactly one key-management service (KMS) provider, built from that plane’s own local deploy configuration:
- Enclave KMS. The customer’s or enclave operator’s own key store. It
guards the sovereign rows in the enclave database, the per-org secret keys,
and the enclave’s attestation (SVID, SPIFFE Verifiable Identity Document) CA
signing key. Only enclave components
dial it. In the dev cluster it is the
dvx-enclave-kmsVault pod in thedvx-enclavenamespace. - Controlplane KMS. The platform operator’s own key store. It guards
controlplane metadata at rest, such as policy rule bodies and config
values. Only the controlplane dials it. In the dev cluster it is the
dvx-cp-kmsVault pod indvx-system.
The package is boundary-safe and encodes no cross-plane path. Each plane
constructs its own
kms.Provider from DVX_KMS_PROVIDER and DVX_KMS_CONFIG on its own dvx serve
component. The controlplane never obtains the enclave KMS endpoint or a
credential to it, so a compromised
controlplane cannot ask the enclave KMS to unwrap anything.
The dev cluster makes the isolation visible: the two Vault servers run in different namespaces, and a NetworkPolicy on each restricts ingress to its own namespace, so the controlplane’s pods cannot reach the enclave’s Vault and vice versa. In production the two run in different clusters entirely, and a cloud enclave KMS must be the customer’s own account and region, or key material crosses the sovereignty boundary.
What the enclave KMS holds
- Per-org data keys. The enclave encrypts each secret value in its
database under a key named
org-<id>-secrets, one key per org. Because an enclave serves exactly one org, this per-org key is effectively the enclave’s key. - The platform key. Fields with no org coordinate, such as the
platform-floor policy on the controlplane side, use the shared
platform-secretskey. That key is never destroyed during org offboarding. - The SVID CA signing key. The enclave’s attestation CA signs every agent
credential through the KMS as a
crypto.Signer. The private key never leaves the KMS and is never written to disk; only the publicca.crttrust bundle is persisted.
Key generation and storage
The provider generates keys lazily, on first use. When a component encrypts
under an org key that does not exist yet, the provider creates it first: an
aes256-gcm96 Transit key for data keys, an ecdsa-p256 key for the CA.
There is no key ceremony to run and no key file to distribute.
The mechanism underneath is standard envelope encryption:
- The encryptor generates a random per-value DEK (data-encrypting key) and encrypts the plaintext locally with AES-256-GCM.
- The KMS wraps the DEK under the KEK (key-encrypting key). The KMS never receives plaintext.
- The ciphertext row stores the envelope plus a
KeyRefthat names the provider, key, and key version.
An AAD (additional authenticated data) binds every ciphertext to its
logical identity: the org, agent or group, and secret name, or for sealed
inference-audit bodies, the org, agent, request id, and form. A stolen
(ciphertext, key_ref) pair fails the authentication check when replayed
onto any other row, agent, or org. Decrypt is fail-closed: an unknown
envelope version, a truncated envelope, or an AAD mismatch returns an error,
never plaintext.
Custody boundaries in the enclave
Four enclave components hold the enclave Crypter, each with a bounded use:
- The broker owns the enclave database and the enclave KMS binding. It encrypts and decrypts sovereign secret values.
- The llmproxy holds a
Crypterso it can seal inference-audit conversation bodies before they reach the audit store. Each record uses a fresh random DEK, so a passive memory scrape of the process yields no standing key over the stored corpus. - The network gateway optionally holds a
Crypterfor the same reason: sealing captured request headers and bodies on mediated egress. Without a configured enclave KMS the gateway captures nothing, so this is a capability that an operator adds deliberately. - The imds daemon holds a
Crypterfor the SVID CA signing path, and for the sovereign store when it is deployed with its own database. In the dev deployment the broker owns the store, so the imds uses itsCrypterfor signing only.
A reader who needs the plaintext of a sealed audit body runs
dvx internal audit-reveal
against the enclave KMS. Decryption is a separate, KMS-audited forensic
operation, not part of the query path.
Rotation
The KMS versions its data keys. The provider records the key version
in each KeyRef, so a backend-side rotation (Vault Transit rotates a key on
request and keeps old versions decrypting) stays compatible with already
written rows; no dvx code has to migrate ciphertext. The controlplane holds
the agent-operator token signing keys in its own KMS, and that rotation works
the other way around: tokens live for 60 seconds, the JWKS endpoint serves
every stored signing key, and an unexpired token keeps verifying through the
rotation overlap.
For service-account key files, rotation is an operator action over the API:
POST /api/v1/orgs/{org}/serviceaccounts/{sa}/keys mints a new key and
returns it exactly once, backed by the audit trail. The platform never holds
those values after delivery. Broker registration keys have no rotation route.
The broker key is minted once at registration and returned once, and
re-running register-broker does not rotate it: the retry hits a Zitadel
username conflict and mints a new enclave id. Replacing a broker key means
registering a new enclave.
Destruction: crypto-shred
Offboarding an org destroys its key. Crypter.DestroyOrgKey destroys the
org-<id>-secrets key material in the provider, and every value wrapped
under it becomes permanently unreadable; the ciphertext rows are tombstones
dropped lazily. The shred is complete the moment the key is gone.
Erasure runs over the operator API in two steps: POST /api/v1/orgs/{org}/erasure
opens a grace window during which nothing is destroyed, and
POST /api/v1/orgs/{org}/erasure/complete executes the irreversible shred.
The route refuses the platform org itself, because the shred would destroy the
operator’s own key material. The
offboarding guide walks the flow.
Providers that cannot delete keys return ErrUnsupportedOperation
instead of performing a shred.