GitOps¶

Application deployments are managed by Flux CD, a GitOps operator that watches a Git repository and continuously reconciles the cluster to match the desired state defined in code.

Branch Strategy¶

Flux watches the flux branch of the Gitea repository — not main. This separation means:

The flux branch is the live state of the cluster. Whatever is committed there is what runs.
The main branch holds infrastructure-as-code (OpenTofu), documentation, and development work — changes there do not trigger cluster reconciliation.
Image automation commits are written directly to flux by Flux's image-automation controller, keeping the branch always current with running image digests.

Bootstrap¶

The reconciliation root is defined in clusters/noah/flux-system/gotk-sync.yaml. This file tells Flux where to find the repository and which branch to watch:

GitRepository: homelab
  url: https://gitea.hdhomelab.com/hd/homelab.git
  branch: flux
  interval: 1m

From there, a root Kustomization reconciles ./flux/clusters/noah, which in turn defines child Kustomization resources for each layer of the stack.

Kustomization Layers¶

The cluster is reconciled in layers, with explicit dependency ordering:

graph LR
    A[flux-system\nFlux controllers] --> B[infra\nCore services]
    A --> C[mon\nMonitoring stack]
    A --> D[apps\nApplications]
    A --> E[tuppr-plans\nTalos & k8s upgrades]

Hold "Alt" / "Option" to enable pan & zoom

Layer	Depends on	Notes
infra	flux-system	Core services: cert-manager, Authentik, ESO config, external-dns, etc.
mon	flux-system	Monitoring stack; reconciles in parallel with infra and apps
apps	flux-system	Applications; reconciles in parallel with infra and mon
tuppr-plans	flux-system	Rolling Talos OS and Kubernetes control plane upgrades

VPA is bootstrapped by OpenTofu

The Vertical Pod Autoscaler controller and its CRDs are installed by the taloser-k8s OpenTofu module during cluster provisioning — before Flux ever runs. This means VPA objects in app manifests reconcile successfully on first apply without any ordering dependency. There is no separate infra-vpa-config kustomization.

Repository Structure¶

flux/
├── clusters/noah/          # Bootstrap entrypoint — Kustomization definitions
│   ├── flux-system/        # Flux components and root GitRepository
│   ├── infra/              # Kustomization pointing to infrastructure/noah
│   ├── apps/               # Kustomization pointing to apps/noah
│   ├── mon/                # Kustomization pointing to monitoring/noah
│   └── tuppr-plans/        # Kustomization for Talos/k8s upgrade plans
├── infrastructure/
│   ├── base/               # Shared core services (included by overlay)
│   └── noah/               # Cluster-specific overlay
├── apps/
│   └── noah/               # Apps organized by category
│       ├── ai/
│       ├── media/
│       ├── home/
│       ├── fin/
│       ├── games/
│       ├── bots/
│       ├── misc/
│       └── devops/
└── monitoring/
    ├── base/               # Shared (Loki, Fluent-bit)
    └── noah/               # Cluster-specific (Prometheus, Grafana, Thanos)

Base / Overlay Pattern¶

Infrastructure and monitoring use a base/overlay split:

infrastructure/base/ — shared core service definitions (cert-manager, Cilium, Authentik, ESO, storage, etc.)
infrastructure/noah/ — cluster overlay that includes ../base and adds cluster-specific resources (Pi-hole, Intel GPU plugin)

This keeps shared configuration in one place while allowing cluster-specific overrides.

App Structure¶

Each app follows a consistent directory pattern:

apps/noah/media/radarr/
├── kustomization.yaml       # Aggregates all resources
├── deployment.yaml          # Deployment + Service
├── pvcs.yaml                # PersistentVolume + PVC (Synology NFS)
├── externalsecret.yaml      # Vault-synced secrets
├── image.yaml               # Image automation resources
└── vpa.yaml                 # VerticalPodAutoscaler

HelmReleases reference HelmRepository sources defined in the infrastructure layer — apps do not define their own chart sources. For storage provisioning, see Storage.

Image Automation¶

Flux automatically updates container image tags in the repository when new versions are published. This uses three resource types chained together:

graph LR
    A[ImageRepository\nPolls registry] --> B[ImagePolicy\nFilters versions]
    B --> C[ImageUpdateAutomation\nCommits update to flux branch]

Hold "Alt" / "Option" to enable pan & zoom

Resource	Role
ImageRepository	Polls the container registry every 24 hours
ImagePolicy	Filters available tags — typically by semver range or regex
ImageUpdateAutomation	Writes the selected tag back into the YAML file and commits to `flux`

Note

A single ImageUpdateAutomation (flux-image-automation in flux-system) runs every 48 hours, scans the entire ./flux tree, and batches all pending image updates into one commit. Each app defines its own ImageRepository and ImagePolicy, but none carry their own automation. Automated commits use fluxcdbot@flux.noreply.gitea.hdhomelab.com as the author.

Secret Management¶

Secrets are managed via Vault and synced into Kubernetes using the ExternalSecrets Operator. See Security for details.

Notifications¶

Flux sends reconciliation events to a Telegram group. Alerts cover all GitRepository, Kustomization, and HelmRelease resources at info severity — meaning both successes and failures are reported.

Webhook for instant reconciliation

A webhook receiver (flux-webhook.hdhomelab.com) accepts push events from Gitea and triggers immediate reconciliation on the flux-system GitRepository, bypassing the 1-minute polling interval. A git push to the flux branch reconciles the cluster within seconds.

GitOps Dashboard¶

Headlamp (headlamp.hdhomelab.com) runs the Flux plugin, which adds a dedicated Flux section to the UI. It provides visibility into the full GitOps state without needing kubectl or the flux CLI:

View	What it shows
Kustomizations	Reconciliation status, last applied revision, any errors
HelmReleases	Chart version, values, upgrade history
GitRepositories	Sync status, last fetched commit
ImageRepositories	Last scanned tags
ImagePolicies	Selected tag, filter rule
ImageUpdateAutomations	Last commit written to `flux` branch

Tip

The Flux plugin is the fastest way to check why a resource isn't reconciling — it surfaces the same status conditions as flux get but with diff views and drill-down into dependent resources.