CRDs and upgrades¶

VirtFoundry keeps its state in virtfoundry.io custom resources: tenants, instances, networks, templates, VKS clusters and more. The CRDs define those kinds, so how they are installed and upgraded matters.

Deleting a CRD deletes its data

Removing a CRD removes every custom resource of that kind. Deleting tenants.virtfoundry.io deletes all tenants. Never delete these CRDs to "reset" an install.

The problem with Helm¶

Helm installs files in a chart's crds/ directory on the first install only. It does not upgrade or delete them. A new field in a CRD never reaches a cluster through helm upgrade, and the API server then drops the field or rejects the resource.

The `virtfoundry-crds` chart¶

All 16 CRDs live in one chart, charts/virtfoundry-crds, in templates/ so that helm upgrade updates them. Each one is protected:

Protection	Stops
`helm.sh/resource-policy: keep`	`helm uninstall` deleting the CRD
`argocd.argoproj.io/sync-options: Prune=false,Delete=false`	Argo CD pruning the CRD, or deleting it with the Application

The CRD files are copied verbatim from their sources of truth, so they are never parsed as templates:

CRDs	Source of truth
15 platform CRDs	`virtfoundry/operator`, `config/crd/bases` (generated by controller-gen)
`VKSCluster`	`virtfoundry/vks`, `config/crd/bases`

CI fails when charts/virtfoundry-crds/manifests drifts from those sources (make verify-crds-chart). To refresh after a CRD change upstream:

make sync-crds-chart   # copies from operator and vks main
git diff charts/virtfoundry-crds/manifests

Install order¶

From 0.11.0, the operator and VKS charts no longer ship CRDs. Install the CRD chart first, as a separate release:

helm install virtfoundry-crds virtfoundry/virtfoundry-crds \
  --version 0.12.0 -n virtfoundry-system --create-namespace

helm install virtfoundry-operator virtfoundry/virtfoundry-operator \
  --version 0.12.0 -n virtfoundry-system

Releases up to 0.10.x

The 0.10.x operator and VKS charts still ships its own crds/ directory, so with 0.10.x the install in Installation is unchanged: the operator chart installs the CRDs and you upgrade them by hand (below). If you want the CRD chart with a 0.10.x operator chart, install it with --skip-crds, otherwise Helm reports that the CRDs already exist.

Upgrading CRDs with the 0.10.x operator chart¶

Apply the CRDs from the new chart version before upgrading the releases. Server-side apply is required: the CRDs are too large for the client-side annotation.

helm pull virtfoundry/virtfoundry-operator --version <new> --untar
kubectl apply --server-side --force-conflicts -f virtfoundry-operator/crds/
helm upgrade virtfoundry-operator virtfoundry/virtfoundry-operator --version <new> -n virtfoundry-system

The VKSCluster CRD is part of the virtfoundry-vks chart templates, so helm upgrade of that chart updates it.

Adopting existing CRDs into the CRD chart¶

For a cluster that already has the CRDs (installed by the operator chart), tell Helm the new release owns them. Do this once, before the first helm install of virtfoundry-crds:

for crd in $(kubectl get crd -o name | grep '\.virtfoundry\.io$'); do
  kubectl annotate "$crd" meta.helm.sh/release-name=virtfoundry-crds \
    meta.helm.sh/release-namespace=virtfoundry-system --overwrite
  kubectl label "$crd" app.kubernetes.io/managed-by=Helm --overwrite
done

This changes metadata only. No CRD content changes and no custom resource is touched. Then install virtfoundry-crds and the operator chart with --skip-crds.

Rehearse the migration¶

Rehearse on a throwaway cluster, never on one you care about. Only the CRDs and Helm are exercised, so KubeVirt is not needed.

Result (2026-10-06). Run against a real kube-apiserver 1.36.2 (envtest, local, no nodes) with the published operator chart 0.10.0, this repository's virtfoundry-crds and the operator chart without crds/:

Check	Result
Operator chart 0.10.0 installs the CRDs	15 CRDs (`VKSCluster` comes from the VKS chart)
`virtfoundry-crds` installed without adopting	Fails: `exists and cannot be imported into the current release: invalid ownership`
After adopting, install `virtfoundry-crds`	16 CRDs, existing Tenant untouched
Upgrade the operator to the chart without `crds/`	CRDs and Tenant untouched
`helm upgrade` of `virtfoundry-crds` with a changed schema	New field appears, and reverts on the next upgrade (changes propagate)
`helm uninstall virtfoundry-crds`	All 16 CRDs and the Tenant remain

This is now automated: the Chart e2e workflow (.github/workflows/chart-e2e.yaml, script scripts/ci/e2e-charts.sh) runs the fresh install, the umbrella install and this migration on a Kind cluster for every change to charts/. It covers the same checks, and the install flow with the new chart layout. Not covered: the Argo CD behaviour (Prune/Delete options and sync waves), which needs an Argo instance, and a real KubeVirt install.

To repeat the migration by hand on a Linux host with Kind:

kind create cluster --name crd-rehearsal

# 1. Today's layout: the operator chart (0.10.0) installs the CRDs
helm install virtfoundry-operator virtfoundry/virtfoundry-operator --version 0.10.0 \
  -n virtfoundry-system --create-namespace
kubectl apply -f - <<'EOF'
apiVersion: virtfoundry.io/v1alpha1
kind: Tenant
metadata: { name: rehearsal }
spec: { name: Rehearsal, slug: rehearsal }
EOF
kubectl get tenants.virtfoundry.io rehearsal

# 2. Adopt the CRDs (metadata only), then install the CRD chart
for crd in $(kubectl get crd -o name | grep '\.virtfoundry\.io$'); do
  kubectl annotate "$crd" meta.helm.sh/release-name=virtfoundry-crds \
    meta.helm.sh/release-namespace=virtfoundry-system --overwrite
  kubectl label "$crd" app.kubernetes.io/managed-by=Helm --overwrite
done
helm install virtfoundry-crds ./charts/virtfoundry-crds -n virtfoundry-system

# 3. Move the operator to the chart without crds/ (from the operator repository)
helm upgrade virtfoundry-operator ./charts/virtfoundry-operator -n virtfoundry-system

# 4. Uninstalling the CRD release must keep the CRDs and the data
helm uninstall virtfoundry-crds -n virtfoundry-system
kubectl get crd | grep -c '\.virtfoundry\.io'      # expect 16 (15 + VKSCluster)
kubectl get tenants.virtfoundry.io rehearsal         # expect it to still exist

kind delete cluster --name crd-rehearsal

Pass criteria: the Tenant survives every step and all 16 CRDs remain after step 4. The Tenant spec needs name and slug (checked against the 0.10.0 CRD). Record the result in the pull request that removes crds/.

Argo CD¶

Set ServerSideApply=true on the Application that installs the CRDs (large CRDs overflow the client-side annotation).
Install the CRD Application first, with an earlier sync wave than the operator.
The Prune=false,Delete=false option is already on every CRD, so removing the Application leaves them in place.

What this does not solve¶

Destructive schema changes (removing a field, changing a type) still need a migration plan and, for stored objects, a conversion step. The chart only delivers the new schema.
Order matters on upgrade: apply the CRDs first, then the controllers that use new fields.