CI / Jenkins¶

Jenkins is the self-hosted CI system for the homelab. It runs in the devops namespace on the noah cluster, managed by Flux, and integrates with the self-hosted Gitea instance for source, webhooks, and the container registry.

URL — jenkins.hdhomelab.com
Source org — gitea.hdhomelab.com/cicd
Namespace — devops
Flux manifests — flux/apps/noah/devops/jenkins/

Architecture¶

graph LR
    subgraph gitea ["Gitea (gitea.hdhomelab.com)"]
        G1[cicd org repos]
        G2[jenkins-library]
        G3[jenkins-agents]
        G4[Container Registry]
    end

    subgraph k8s ["Kubernetes — devops namespace"]
        J[Jenkins controller]
        A1[Agent pod\ndocker / dind]
        A2[Agent pod\nmaven / node / python]
        A3[Agent pod\nsemantic-release]
    end

    G1 -->|webhook + multibranch scan| J
    G2 -->|shared library| J
    J -->|spawns| A1
    J -->|spawns| A2
    J -->|spawns| A3
    A1 -->|push image| G4
    A3 -->|push image| G4

Hold "Alt" / "Option" to enable pan & zoom

Deployment¶

Jenkins is installed via the official Helm chart (5.x). Configuration-as-Code (JCasC) drives all runtime config — no manual Jenkins UI changes persist across pod restarts.

Setting	Value
Chart	`jenkins/jenkins 5.*`
Java heap	`-Xms512m -Xmx2048m`
Controller resources	200m CPU req · 256Mi–3Gi memory
Storage	PVC `jenkins` (persistent home)
Node	`worker-2a` (pinned via `nodeSelector`)
Timezone	`America/New_York`

Authentication¶

Login is handled by Authentik OIDC — the local admin account is disabled. Groups from Authentik map to Jenkins permissions:

Authentik group	Jenkins permission
`jenkins_admin`	Overall/Administer
`authenticated`	Overall/Read
`jenkins_user`	Job/Build

Secrets¶

Secrets are injected via ExternalSecrets (Vault) and mounted as Kubernetes secrets. JCasC references them by variable substitution (${secret-key}):

Secret	Contents
`jenkins-admin`	Admin username + password
`jenkins-oidc`	Authentik OIDC client ID + secret
`jenkins-gitea`	Gitea PAT + registry username/password

Gitea Integration¶

Jenkins automatically discovers all repositories in the cicd Gitea org via a Job DSL-provisioned organization folder:

Branch discovery — all branches
PR discovery — head + merged
Orphan cleanup — items pruned after 14 days / 40 builds
Fallback scan — daily (in addition to webhook-triggered scans)

Webhooks are managed by the Gitea Jenkins plugin — no manual webhook setup needed when adding new repos to the cicd org.

Agent Templates¶

All build agents are ephemeral Kubernetes pods spawned on demand and deleted after the job completes. The default template is docker.

docker (default)mavennodepythonsemantic-release

Docker-in-Docker setup. Two containers share a pod:

Container	Image	Role
`docker`	`docker:28-cli`	Runs `docker build / push` commands
`dind`	`docker:28-dind`	Docker daemon (privileged, `overlay2`)

Communication: DOCKER_HOST=tcp://localhost:2375 (TLS disabled for simplicity).

Resources (dind): 500m–2 CPU · 1–4Gi memory.

maven:3-eclipse-temurin-17 — Java/Maven builds.

node:20-alpine — Node.js builds.

python:3 — Python jobs. 400m–1 CPU · 512Mi–1Gi memory.

gitea.hdhomelab.com/cicd/jenkins-semantic-release:latest — custom image from the jenkins-agents repo. Includes semantic-release + Gitea release plugin. Requires gitea-registry pull secret.

emptyDir volumes on Talos

The docker template mounts an emptyDirVolume at /var/lib/docker for the dind daemon. This is the only place emptyDir is used, justified because the Docker layer cache is genuinely ephemeral and dind requires a writable, non-overlapping mount.

Shared Library¶

The shared library lives in gitea.hdhomelab.com/cicd/jenkins-library (main branch). It is globally registered in JCasC and must be explicitly loaded in Jenkinsfiles:

@Library('shared@main') _

`dockerWait`¶

Polls the Docker daemon until it is ready. Required before any docker command when using the DinD agent, because the daemon takes a few seconds to start after the pod is scheduled.

dockerWait(timeoutSeconds: 90)  // default: 60s, checks 'docker version' every 1s

`stepBuild`¶

Builds and pushes a Docker image to the Gitea container registry.

Tag resolution order: explicit tag → version file → git short SHA → BUILD_NUMBER → latest.

def result = stepBuild(
    registry:      'gitea.hdhomelab.com',
    credentialsId: 'gitea-registry',
    repository:    'cicd/my-app',       // required
    dockerfile:    'Dockerfile',        // default
    context:       '.',                 // default
    buildArgs:     [BUILD_ENV: 'prod'], // optional
    pushLatest:    true,                // default
    additionalTags: ['1.2.3'],          // optional
    skipPush:      false,               // default
)
// result: [image, tag, fullImage, pushed]

If a non-latest tag already exists in the registry, the step pauses and prompts the user to confirm before overwriting (confirmOverwrite: true by default).

`standardAppPipeline`¶

Thin wrapper around stepBuild for the common case: one Build stage on the docker agent. Automatically detects PRs (env.CHANGE_ID) and skips push on PR builds, tagging them pr-<number> instead.

@Library('shared@main') _

standardAppPipeline(
    registry:      'gitea.hdhomelab.com',
    credentialsId: 'gitea-registry',
    repository:    'cicd/my-app',
)

Custom Agent Images¶

Custom agent images are maintained in gitea.hdhomelab.com/cicd/jenkins-agents. The repo's Jenkinsfile auto-discovers all agent subdirectories via find:

jenkins-agents/
└── semantic-release/
    └── Dockerfile       ← produces gitea.hdhomelab.com/cicd/jenkins-semantic-release

Build behavior:

All branches: build only (validate Dockerfile builds)
main branch: push :latest + 7-char git SHA to the Gitea registry

Adding a new agent is as simple as creating a new subdirectory with a Dockerfile — no pipeline config changes required.

`jenkins-semantic-release`¶

FROM node:20-slim
RUN npm install -g \
    semantic-release \
    @semantic-release/commit-analyzer \
    @semantic-release/release-notes-generator \
    @semantic-release/changelog \
    @semantic-release/git \
    @saithodev/semantic-release-gitea

Bundles semantic-release with the Gitea release adapter for automated changelog and release management from CI.

Monitoring¶

Jenkins exposes Prometheus metrics at /prometheus. The pod carries scrape annotations so the Prometheus stack picks it up automatically:

prometheus.io/scrape: "true"
prometheus.io/path: /prometheus
prometheus.io/port: "8080"

The prometheusConfiguration JCasC block disables disk usage collection (expensive) while keeping all other metrics enabled.