GitOps with Flux¶

Flux CD is the continuous-delivery engine for the platform EKS clusters. It reconciles the desired cluster state from the platform-infra Git repository, decrypts SOPS secrets, and automatically bumps container image tags as new releases are published. This page documents the Flux GitOps model: the per-cluster bootstrap roots, the ordering guarantees, the apps/<app>/{base,overlays} convention, image automation, and the SOPS decryption wiring — plus the prod-vs-staging differences.

Flux version in use: v2.6.1 (see the # Flux Version header in gotk-components.yaml).

For the narrative companion to this reference, see kubernetes/docs/gitops-apps.md and the docs/runbooks/flux-bootstrap.md.

The core idea: infra wires GitOps, services own their manifests¶

platform-infra does not hold long-lived copies of application Deployments. Instead, each app directory under kubernetes/apps/<app>/ holds only Flux Custom Resources:

GitRepository — points at the application's own repo (e.g. admin-backend, ingest-driver).
Kustomization — a Flux Kustomization whose spec.path resolves inside the app repo (./k8s/overlays/<env>), not under platform-infra.
ImageRepository / ImagePolicy / ImageUpdateAutomation — the image-automation triad that scans GHCR and writes new tags back to the app repo.

The real Kubernetes manifests (Deployments, Services, Ingresses, ConfigMaps) live in each service's own repository under k8s/overlays/staging and k8s/overlays/prod. Engineers change a workload by merging to the branch Flux tracks for that app — no platform-infra PR is required unless the wiring changes (branch, interval, health checks, image policy, or adding a new app).

Exceptions to the split

The split is now uniform for applications. The former exception — a legacy inline capture manifest tree under kubernetes/overlays/staging-cpt-aws/apps/… — was removed in commit 8567811c (see Prod vs staging). Infrastructure services and controllers under kubernetes/infrastructure/ are, by design, still defined in this repository.

Cluster roots — the `flux bootstrap --path` targets¶

Each cluster is bootstrapped once against a single overlay directory. That directory's kustomization.yaml is the root of the dependency graph Flux reconciles.

Cluster	Bootstrap path	Git ref tracked	SOPS secret (root)
Production (`prod-eks`)	`kubernetes/overlays/prod-cpt-aws`	`semver: ">=1.0.0"` (version tags `vX.Y.Z`)	`sops-age`
Staging (`staging-eks`)	`kubernetes/overlays/staging-cpt-aws`	`branch: main`	`sops-age`

Staging tracks main; there is no office overlay in this repo

gotk-sync.yaml for staging points at branch: main — the dedicated staging branch was retired after it drifted behind. An office-cpt-onprem cluster is mentioned in the README and the Flux bootstrap runbook, but no such overlay exists under kubernetes/overlays/ today — only prod-cpt-aws and staging-cpt-aws.

Bootstrap command (from docs/runbooks/flux-bootstrap.md):

flux bootstrap github \
  --owner=example-org \
  --repository=platform-infra \
  --branch=main \
  --path=kubernetes/overlays/<environment> \
  --personal

Pre-bootstrap step differs by cluster type

The EKS clusters (prod-cpt-aws, staging-cpt-aws) get their CNI and kube-proxy from managed EKS add-ons provisioned by Terraform (aws_eks_addon.vpc_cni, aws_eks_addon.kube_proxy in terraform/aws/_modules/eks/_eks-addons.tf), so there is no CNI bootstrap ordering problem to solve on EKS.

A common/ overlay exists only for staging-cpt-aws, and today it contains cert-manager only. The Calico ordering described in the Flux bootstrap runbook applies to the Ansible-provisioned on-prem path, not to EKS. See that runbook for the current per-cluster detail.

The `flux-system` directory¶

Each root contains a flux-system/ subdirectory generated by flux bootstrap (marked # This manifest was generated by flux. DO NOT EDIT.):

File	Purpose
`gotk-components.yaml`	The Flux controller install (source, kustomize, helm, notification, image-reflector, image-automation controllers), v2.6.1
`gotk-sync.yaml`	The root `GitRepository` + `Kustomization` that point Flux back at this repo/path
`kustomization.yaml`	Bundles the two `gotk-*` files

The root GitRepository uses SSH (ssh://git@github.com/example-org/platform-infra, secret flux-system) — distinct from the HTTPS + github-app credentials the per-app GitRepository objects use. The root Kustomization reconciles ./kubernetes/overlays/<env> with prune: true, interval: 5m0s, and top-level SOPS decryption via secretRef.name: sops-age.

Prod (semver) vs staging (branch) — the only functional difference in the two gotk-sync.yaml files:

# prod-cpt-aws/flux-system/gotk-sync.yaml
  ref:
    semver: ">=1.0.0"     # production tracks version tags; rollback = re-tag
# staging-cpt-aws/flux-system/gotk-sync.yaml
  ref:
    branch: main          # the staging cluster root tracks main (dedicated staging branch retired)

Reconciliation ordering¶

The root kustomization.yaml lists resources in a deliberate order so that controllers and cluster-wide prerequisites exist before any app Kustomization reconciles. (Kustomize does not enforce apply order between resources, but the shared prerequisites — namespaces, pull secrets, external-secrets controller, priority classes — must be present for app manifests and SOPS decryption to succeed.)

# kubernetes/overlays/prod-cpt-aws/kustomization.yaml (abridged)
resources:
  - flux-system                              # 1. Flux controllers (already installed)
  # Infra — secrets/priority-classes/shared namespaces BEFORE app wiring
  - ../../infrastructure/controllers         # 2. external-secrets, external-dns, cloudflared, minio-operator
  - ../../infrastructure/configs             # 3. github-app, ghcr-image-pull, priority-classes + controller patches
  # Apps — Flux CRs only (manifests live in each app repo)
  - ../../apps/backend/base
  - ../../apps/backend/overlays/prod
  - ../../apps/public-api/base
  - ../../apps/public-api/overlays/prod
  # …camera-console, driver, status, utils…
  # Services — Flux dirs under each service's overlays/<env>/flux
  - ../../infrastructure/services/observability/overlays/prod/flux
  - ../../infrastructure/services/github-runner/overlays/prod/flux
  - ../../infrastructure/services/netshoot

The tiers:

flux-system — bootstrap controllers.
infrastructure/controllers — shared controllers: external-secrets, external-dns, cloudflared, minio-operator (cert-manager is present but disabled — "not compatible with AWS Certificate Manager & ALB").
infrastructure/configs — github-app,ghcr-image-pull,priority-classes.yaml, plus JSON-patch resource limits applied to every Flux controller Deployment.
apps/… — Flux CRs only (base + env overlay per app).
infrastructure/services/… — in-cluster infra services whose prod Flux Kustomization lives under overlays/prod/flux/ (satisfies Kustomize load restrictions when the root runs kubectl kustomize).

Blast radius

Changing the order or removing infrastructure/controllers / infrastructure/configs can stop Flux decrypting secrets or applying shared prerequisites — treat edits there as cluster-wide. Individual apps can be removed from the resources: list without touching flux-system. Rollback: revert the overlay commit or restore the previous resources: list.

The `apps/<app>/{base,overlays/<env>}` convention¶

Every GitOps-managed app follows the same Kustomize base/overlay shape. base/ carries env-independent Flux sources and any shared cluster objects; overlays/<env>/ carries the per-env Flux Kustomization, GitRepository (branch), ImagePolicy (tag pattern), and ImageUpdateAutomation.

Canonical layout (backend, wiring the external admin-backend repo):

kubernetes/apps/backend/
├── base/
│   ├── kustomization.yaml
│   ├── imagerepository.yaml     # ImageRepository → ghcr.io/example-org/backend
│   ├── namespace.yaml           # creates the shared `ingest` namespace
│   ├── externalsecret.yaml      # ghcr-credentials pull secret (ExternalSecret)
│   └── serviceaccount.yaml      # default SA in ingest with imagePullSecrets
└── overlays/
    ├── prod/
    │   ├── kustomization.yaml
    │   ├── flux-kustomization.yaml     # Flux Kustomization → path ./k8s/overlays/prod IN admin-backend
    │   ├── gitrepository.yaml          # branch: main
    │   ├── imagepolicy.yaml            # ^v[0-9]+.[0-9]+.[0-9]+$  (stable)
    │   └── imageupdateautomation.yaml  # writes back to admin-backend main
    └── staging/
        └── …                           # branch: staging, -rc tag pattern, writes back to staging

base kustomization¶

# kubernetes/apps/backend/base/kustomization.yaml
resources:
  - imagerepository.yaml
  - namespace.yaml       # ingest namespace
  - externalsecret.yaml  # ghcr-credentials (shared by driver too)
  - serviceaccount.yaml

Not every app carries all base objects. driver/base holds only imagerepository.yaml — it reuses the ingest namespace, ghcr-credentials ExternalSecret and default ServiceAccount created by backend/base. This is why backend must be listed before / alongside driver in the root.

The Flux `Kustomization` (points into the app repo)¶

# kubernetes/apps/backend/overlays/prod/flux-kustomization.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: admin-backend
  namespace: flux-system
  labels:
    app.kubernetes.io/sops: "enabled"   # opts into the cluster-wide SOPS decryption patch
spec:
  interval: 5m
  path: ./k8s/overlays/prod             # resolved INSIDE the admin-backend clone
  prune: true
  sourceRef:
    kind: GitRepository
    name: admin-backend
    namespace: flux-system
  timeout: 5m
  wait: true
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: admin-backend
      namespace: ingest

The per-env `GitRepository`¶

# kubernetes/apps/backend/overlays/prod/gitrepository.yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: admin-backend
  namespace: flux-system
spec:
  interval: 5m
  provider: github
  ref:
    branch: main            # prod → main; staging overlay → branch: staging
  secretRef:
    name: github-app        # HTTPS via the shared GitHub App credentials
  url: https://github.com/example-org/admin-backend

App inventory¶

Every app under kubernetes/apps/ and how it is wired. "SOPS" = the Flux Kustomization carries the app.kubernetes.io/sops: enabled label (still SOPS-based); apps without it have migrated to ExternalSecret (AWS Secrets Manager).

App dir	Flux `Kustomization` name	App repo (`GitRepository.url`)	Target ns	SOPS?	Notes
`backend`	`admin-backend`	`admin-backend`	`ingest`	Yes	Creates shared `ingest` ns, `ghcr-credentials`, default SA; `wait`+healthCheck
`driver`	`driver`	`ingest-driver`	`ingest`	Yes	base = ImageRepository only; reuses `ingest` prerequisites
`public-api`	`public-api`	`public-api`	`dashboard`	No (migrated)	ImagePolicy bounded `>=1.0.0 <2.0.0`; creates `dashboard` ns
`camera-console`	`camera-console`	`camera-console`	`camera-console`	No (ExternalSecret)	No `wait`/healthCheck; `timeout: 2m`
`status`	`status-page`	`status-page`	`status`	Yes	`wait`+healthCheck pattern
`utils`	one per component (see below)	`platform-utils` (shared)	`utils`	No	Multi-component; shared GitRepository

`utils` — multi-component with a shared `GitRepository`¶

utils/base declares one shared GitRepository (platform-utils, branch main), the utils namespace, an ExternalSecret and ServiceAccount, then one sub-directory per exporter/job — each with its own ImageRepository. The env overlay (utils/overlays/<env>) lists the components to enable, and each component has its own Flux Kustomization pointing at a path inside platform-utils:

# kubernetes/apps/utils/overlays/prod/capture-exporter/flux-kustomization.yaml
spec:
  path: ./k8s/overlays/prod/capture-exporter   # inside platform-utils
  sourceRef:
    kind: GitRepository
    name: platform-utils

Prod components: camera-image-size-report, distance-cache-cleanup, harddisk-hoover, sonic-stragglers-report, watchlist-log-items, camera-probe-propagator, capture-exporter, mikrotik-wireguard-exporter, router-fleet-resolver, mktxp. Staging enables only distance-cache-cleanup, harddisk-hoover, watchlist-log-items.

mktxp has no image automation

utils/overlays/prod/mktxp contains only a flux-kustomization.yaml (no ImagePolicy / ImageUpdateAutomation) — it is deployed from platform-utils but its image tag is not auto-bumped by Flux.

Flux image automation¶

Each app (except mktxp) runs the full image-automation triad. The flow:

ImageRepository (in base/) scans a GHCR image on an interval using ghcr-credentials:

apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
spec:
  image: ghcr.io/example-org/backend
  interval: 5m
  provider: generic
  secretRef:
    name: ghcr-credentials

ImagePolicy (in overlays/<env>/) selects the newest tag matching the env's pattern:

Env Pattern Range

prod ^v[0-9]+\.[0-9]+\.[0-9]+$ >=1.0.0 (public-api: >=1.0.0 <2.0.0)

staging ^v[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$ >=1.0.0-rc.0 (public-api: >=0.1.0-rc.0)

So prod tracks stable releases and staging tracks release candidates for the same image.
ImageUpdateAutomation writes the selected tag back to the app repo using the Setters strategy, driven by inline # {"$imagepolicy": "flux-system:<policy>"} markers next to each image field. It commits as platform-bot <actions@users.noreply.github.com> with a [ci skip] message on the tracked branch (main for prod, staging for staging).

# overlays/prod/imageupdateautomation.yaml (abridged)
spec:
  git:
    commit:
      author: { email: actions@users.noreply.github.com, name: platform-bot }
      messageTemplate: |
        chore: update admin-backend image to {{ range .Updated.Images }}{{ println . }}{{ end }}
        [ci skip]
    push: { branch: main }
  update:
    path: "./k8s/overlays/prod"
    strategy: Setters

The image-reflector-controller (which scans registries) is given 512Mi memory in infrastructure/configs — the largest allocation of any Flux controller, reflecting the tag-scanning workload.

SOPS decryption¶

There are two distinct SOPS layers:

Root layer — the top-level flux-system Kustomization in gotk-sync.yaml decrypts with provider: sops, secretRef.name: sops-age. This covers SOPS-encrypted files reconciled directly from platform-infra.

Per-app patch — the root kustomization.yaml applies a strategic-merge patch that injects SOPS decryption into every Flux Kustomization labelled app.kubernetes.io/sops=enabled, using secretRef.name: sops-keys:

patches:
  - patch: |
      apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
      kind: Kustomization
      metadata:
        name: all
      spec:
        decryption:
          provider: sops
          secretRef:
            name: sops-keys
    target:
      kind: Kustomization
      labelSelector: app.kubernetes.io/sops=enabled

This is how apps whose manifests still contain SOPS-encrypted secret.yaml (in their own repos) get those secrets decrypted at reconcile time. Apps carrying the label today: backend/admin-backend, driver, status. Apps that have dropped it (migrated to ExternalSecret via the aws-secrets-manager ClusterSecretStore): public-api, camera-console. public-api is the reference migration — per-env ExternalSecret manifests, no secret.yaml, no SOPS label.

The sops-keys age secret is created out-of-band during bootstrap:

kubectl create secret generic sops-keys \
  --namespace=flux-system \
  --from-file=identity.agekey=<path-to-age-key>

Secret-name mismatch: sops-age vs sops-keys

The root Kustomization (gotk-sync.yaml) references sops-age, while the per-app decryption patch and the bootstrap procedure reference sops-keys. These are two different secret names in the same namespace; if you rotate age keys, make sure the secret name the layer expects actually exists. This inconsistency is called out in Legacy notes.

Do not un-encrypt or .gitignore a committed SOPS secret

Until an app migrates to ExternalSecret, keep its encrypted secret.yaml in the app repo and do not .gitignore it once encrypted. Migrating is a deliberate per-repo change: remove the SOPS label + patch reliance, add an ExternalSecret, and update docs.

Prod vs staging differences¶

Aspect	prod-cpt-aws	staging-cpt-aws
Git tracking (cluster root)	`semver: ">=1.0.0"` (version tags)	`branch: main`
Image policy	stable `vX.Y.Z`	release candidates `vX.Y.Z-rc.N`
App set wired via `../../apps/*`	backend, public-api, camera-console, driver, status, utils	backend, public-api, camera-console, driver, status, utils
Infra services	observability + github-runner + netshoot (via `overlays/prod/flux`)	observability, netshoot, cloudnative-pg, postgres, valkey, openreplay, appsec-foundation, defectdojo, dependency-track, sonarqube, security-integrations. `github-runner` and `minio` are commented out.
external-dns	base (prod) values	patched via `patches/external-dns-staging.yaml` (cross-account Route53, `txtOwnerId: staging-eks`, `domainFilters: staging.cpt.aws.example.net`, IRSA role `444455556666`, assume-role `210987654321`)
capture delivery	Fully migrated — `admin-backend` (+ workers) delivered from the `admin-backend` repo via Flux	Fully migrated, same as prod (the legacy inline tree was removed in `8567811c`)
`minio` service	commented out ("we are not ready for this yet")	commented out

Staging's inline capture tree (removed)¶

Staging previously carried a large inline manifest tree under kubernetes/overlays/staging-cpt-aws/apps/edge-capture/, a remnant of the pre-migration model in which capture workloads were defined here rather than in their own repositories.

That tree was removed in commit 8567811c (chore: remove orphaned inline staging capture manifests). Staging capture workloads are now delivered exactly as prod's are — from the admin-backend and ingest-driver repositories via the backend and driver Flux Kustomizations. The only remaining entry under overlays/staging-cpt-aws/apps/ is aws-load-balancer-controller.

If you still see this tree on disk

Some working copies carry untracked Syncthing sync-conflict files under that path. They are not in Git and are not reconciled by Flux. Confirm against the repository (git ls-tree -r --name-only origin/main kubernetes/overlays/staging-cpt-aws/apps/) rather than the local filesystem before acting on anything found there.

Adding a new GitOps-managed app¶

Add kubernetes/apps/<name>/base/ with at least imagerepository.yaml (and gitrepository.yaml if the source is not shared).
Add kubernetes/apps/<name>/overlays/<env>/ with flux-kustomization.yaml, gitrepository.yaml, imagepolicy.yaml, imageupdateautomation.yaml, matching the existing apps.
Reference ../../apps/<name>/base and ../../apps/<name>/overlays/<env> from the cluster root kubernetes/overlays/<cluster>/kustomization.yaml.
Ensure the app repo contains k8s/overlays/<env> at the path used in spec.path.
For a new app that needs SOPS, add the app.kubernetes.io/sops: enabled label to its Flux Kustomization; prefer ExternalSecret for greenfield apps.

For in-cluster infrastructure services (wired from kubernetes/infrastructure/services/…), production Flux Kustomization manifests live under overlays/prod/flux/ (a flux-kustomization.yaml beside that folder's kustomization.yaml) so the root kubectl kustomize can load them.

Operational quick reference¶

# After bootstrap — verify controllers and reconciliation
flux check
flux get kustomizations
kubectl get pods -n flux-system

# Force a reconcile of one app
flux reconcile kustomization admin-backend -n flux-system --with-source

# See image automation state
flux get image repository
flux get image policy
flux get image update

# Recover a stuck Kustomization finalizer
kubectl patch kustomization <name> -n flux-system \
  -p '{"metadata":{"finalizers":null}}' --type=merge

Rolling back an application requires suspending image automation first

Reverting an application image tag by hand does not hold on its own. The prod ImagePolicy selects the highest semver tag (range: '>=1.0.0') and ImageUpdateAutomation re-scans every 5 minutes, so a hand-edited tag is overwritten within roughly five minutes. Suspend image automation before reverting.

Full procedure, per scenario: Rollback procedures.

Two independent rollback surfaces

Rolling back the cluster root and rolling back an application are different operations, and the distinction matters under pressure:

Cluster root — which revision of platform-infra a cluster reconciles. Prod tracks semver version tags; staging tracks the main branch. Both are set in kubernetes/overlays/<cluster>/flux-system/gotk-sync.yaml.
Application — which image tag an app runs. Governed by the per-app GitRepository, ImagePolicy and ImageUpdateAutomation under kubernetes/apps/, and subject to the automation caveat above.

Legacy notes¶

See the structured findings returned with this page. In summary, the following were observed while documenting this subsystem:

Orphaned staging inline capture tree — kubernetes/overlays/staging-cpt-aws/apps/** (capture manifests, resources-patch.yaml, local apps/kustomization.yaml with its images: block) is not referenced by the staging root kustomization.yaml; superseded by the ../../apps/* Flux-CR wiring.
Commented-out edge-capture/kustomization.yaml — most resources and the entire patches: block are commented, with "check base for duplicates before uncommenting" notes.
Orphaned aws-load-balancer-controller — present under staging apps/ but commented out in apps/kustomization.yaml, and the whole apps/ dir is unreferenced.
Orphaned staging common/ — common/kustomization.yaml references the disabled cert-manager controller and is not referenced by the root.
sops-age vs sops-keys — the root Kustomization decrypts with sops-age; the per-app patch and bootstrap doc use sops-keys.
Committed generated files — gotk-components.yaml / gotk-sync.yaml are Flux-generated (DO NOT EDIT); expected for flux bootstrap, noted for completeness.

Env	Pattern	Range
prod	`^v[0-9]+\.[0-9]+\.[0-9]+$`	`>=1.0.0` (public-api: `>=1.0.0 <2.0.0`)
staging	`^v[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$`	`>=1.0.0-rc.0` (public-api: `>=0.1.0-rc.0`)