Daevix Docs

Self-hosted controlplane

View as Markdown Edit on Tangled
This page is the engineering document, published as-is from the repository. It is reconciled with the code on every change, so it is denser than a written guide and its section references point at the source rather than at this site. docs/self-hosted-controlplane.md · b93b416a

Status: documented + smoke-tested against the daevix-dev cluster.

The controlplane is deployable SaaS or self-hosted. Self-hosting it is the top of the BYOK spectrum (see architecture.md, “The BYOK spectrum”):

TierAnswerWho holds the key
1operator-CMK at rest (bd_2_0-k5t.7)operator
2BYO-enclave — run your own enclaveyou (sovereign data)
3BYO-controlplane — self-host the whole management planeyou (everything)

A regulated org that will not let the operator hold any key of theirs runs the controlplane themselves and points its operator-CMK at their own KMS. Nothing changes in the binary or the invariants — a self-hosted controlplane still never receives sovereign data (invariants 1/2), and it still mints identity as the sole authority (invariant 5). The only differences are who runs it and whose KMS guards its metadata at rest.

What you deploy

The same manifests the SaaS deploy uses, in deploy/controlplane/:

ManifestComponent
namespace.yamlthe dvx-system namespace
postgres.yamliam-postgres — the controlplane database (own it; holds no sovereign data)
zitadel.yamlZitadel — identity/federation (the CP trusts exactly this one issuer)
vault.yamldvx-cp-kms — the operator-CMK Vault (or omit and point at your own KMS)
controlplane.yamlthe dvx-controlplane Deployment + Service

The controlplane binary is one image (localhost/dvx:dev in dev; your registry in production), run as dvx serve controlplane.

The one knob that makes it “yours”: the operator-CMK

controlplane.yaml wires the operator-CMK via DVX_KMS_PROVIDER + DVX_KMS_CONFIG (bd_2_0-k5t.7). Self-hosting means these point at your KMS. The controlplane constructs this provider itself, controlplane-side only — it never dials an enclave or a customer KMS, and the token is read from a mounted file, never baked into the manifest.

Option A — the bundled Vault (simplest; dev and small self-hosts). Deploy vault.yaml and use it as-is. This is what daevix-dev runs:

- name: DVX_KMS_PROVIDER
  value: vault
- name: DVX_KMS_CONFIG
  value: |
    {"address":"http://dvx-cp-kms.dvx-system.svc.cluster.local:8200","token_path":"/etc/dvx/kms/token","mount_path":"transit"}

Option B — your own external KMS (production self-host). Skip vault.yaml and point DVX_KMS_CONFIG at your existing Vault (or a future cloud-KMS provider). Give the controlplane a scoped token (Transit encrypt/decrypt/create + key-delete for crypto-shred), not a root token, projected as a file:

- name: DVX_KMS_PROVIDER
  value: vault
- name: DVX_KMS_CONFIG
  value: |
    {"address":"https://vault.corp.example:8200","token_path":"/etc/dvx/kms/token","mount_path":"transit","namespace":"daevix","tls_ca_path":"/etc/dvx/kms/ca.crt"}

Mount the token (and CA) from your own Secret at /etc/dvx/kms (the manifest already mounts a kms-token volume; repoint its secretName). With no KMS configured at all the controlplane still runs, but policy values are stored plaintext — not a BYO-controlplane posture, so a real self-host always sets this.

Deploy steps (daevix-dev-style cluster)

kubectl apply -f deploy/controlplane/namespace.yaml
kubectl apply -f deploy/controlplane/postgres.yaml
kubectl rollout status deploy/iam-postgres -n dvx-system

# Operator-CMK: bundled Vault (Option A), or skip and use your own (Option B).
kubectl apply -f deploy/controlplane/vault.yaml
kubectl rollout status deploy/dvx-cp-kms -n dvx-system

kubectl apply -f deploy/controlplane/zitadel.yaml
kubectl rollout status deploy/zitadel -n dvx-system

# Create the DB + apply migrations (through 00020: IAM coordinates/grants, the
# org KIND enum, the agent-operator token signing key 00019, agent containment
# 00020). Re-runnable; goose tracks the applied version.
#   kubectl port-forward -n dvx-system svc/iam-postgres 55432:5432 &
make iam-migrate

# Apply the controlplane's audit-store (ClickHouse) migrations BEFORE it serves.
# The controlplane fails closed at startup if its audit schema is behind the
# binary (bd_2_0-6p3): a lagging audit DB would otherwise 500 every audit read and
# silently drop audit writes, so `auditstore.New` refuses to start until migrated.
#   kubectl port-forward -n dvx-system svc/controlplane-clickhouse 58000:9000 &
make cp-audit-migrate

kubectl apply -f deploy/controlplane/controlplane.yaml
kubectl rollout status deploy/dvx-controlplane -n dvx-system

# Seed the platform authority in-boundary (SeedPlatform): the platform org
# (kind=platform, a DB-enforced singleton), a data-bearing platform-ops org, and
# the first platform-operator grant for your Zitadel subject. Idempotent.
#   kubectl port-forward -n dvx-system svc/iam-postgres 55432:5432 &
dvx internal bootstrap \
  --database-dsn "postgres://postgres:...@127.0.0.1:55432/controlplane?sslmode=disable" \
  --subject <your-zitadel-sub>

The platform-operator role holds only platform-scoped capabilities — it runs the fleet but has no standing authority inside any customer org (there is no super-user; see iam.md §11). Grant org-scoped authority (admin, member, or individual capabilities) per org via the operator API.

The controlplane comes up and logs controlplane at-rest KMS provider ready provider=vault. From here the operator API (with --oidc-issuer + a Zitadel key) authors policy, whose org-sensitive values are encrypted at rest under your KMS, and offboarding (bd_2_0-k5t.8) crypto-shreds an org against that same key.

Smoke test performed

Against the daevix-dev cluster (rootful-podman kind), the self-hosted path was validated end to end:

  • Manifests deploy: kubectl apply of deploy/controlplane/* reconciles; dvx-cp-kms, iam-postgres, zitadel, and dvx-controlplane all reach Ready.
  • Operator-CMK reachable + provider constructs: the controlplane boots against dvx-cp-kms and logs controlplane at-rest KMS provider ready provider=vault (the operator provider built controlplane-side from DVX_KMS_*, token from the projected secret file).
  • Schema applies: goose … up runs cleanly through the then-current schema and is re-runnable as the tree grows (make iam-migrate is idempotent); the down migration reverses it.
  • At-rest behavior: policy values round-trip through encrypt→store→decrypt, a cross-org replayed ciphertext fails closed, and org offboarding crypto-shreds the org key so its rows become undecryptable (unit-verified; see bd_2_0-k5t.7/k5t.8).

Production hardening (beyond dev)

  • KMS: Option B with an external Vault/cloud KMS and a scoped, renewable token — not the dev root token in vault.yaml’s Secret. The renewer in internal/kms/vault keeps a periodic token alive.
  • Database: run iam-postgres HA / managed; it is your data now.
  • TLS + issuer: real certs (not the dev dvx-tls), a hardened Zitadel, and set DVX_OIDC_AUDIENCE.
  • Agent-operator issuer: the controlplane’s own short-lived token issuer (the second operator-API authenticator) needs its PUBLIC issuer URL (DVX_AGENT_TOKEN_ISSUER_URL) set to the externally reachable address, and its signing key rides the same operator KMS (agent_token_key, migration 00019). Its discovery + JWKS endpoints are public by design; pin DVX_OIDC_AUDIENCE so the token class is audience-scoped (see wif.md for federating these tokens into a cloud).
  • Isolation: the dvx-cp-kms-isolation NetworkPolicy already restricts the CMK Vault to same-namespace clients; keep an equivalent policy for an external KMS path.