Skip to content

Router fleet tooling

Two cooperating CronJobs in platform-utils keep the MikroTik router fleet monitored and its AWS Site-to-Site VPN tunnels healthy. Both live in the observability namespace of staging-eks/prod-eks, share one curated inventory and one set of read credentials, and use the same "one credential, at most two connect attempts (api-ssl:8729 then api:8728), no lockout" RouterOS-API pattern.

Component Role Image Schedule
router-fleet-resolver Classifies each router into the authenticated (mktxp) vs unauthenticated (blackbox) tier and publishes both configs ghcr.io/example-org/router-fleet-resolver 17 * * * * (hourly)
router-lifetime-reconciler Audits (and optionally corrects) MikroTik IPsec IKE/child-SA lifetimes + DPD on the AWS VPN peers ghcr.io/example-org/router-lifetime-reconciler 37 * * * * (hourly, offset from the resolver)

Where the code and manifests live

Source and Kubernetes manifests are in platform-utils; platform-infra only wires them into Flux. Source dirs: src/router-fleet-resolver/ and src/router-lifetime-reconciler/; manifests under k8s/base/ and k8s/overlays/prod/.

Shared inputs

Both jobs read the same two mounted inputs, owned by router-fleet-resolver and reused by name in the reconciler:

  • Inventory ConfigMap router-fleet-inventory — generated by kustomize configMapGenerator from the standalone k8s/base/router-fleet-resolver/routers-inventory.yml, mounted at /etc/router-fleet/inventory. This is the single source of truth for which routers are monitored (grouped by client, each with an endpoint and site). Only routers in the inventory are ever scraped — a credential secret alone never adds a target. Routers under the client: orphaned group are kept for a human to re-file or delete but are not scraped.
  • Credentials Secret router-fleet-credentials — one AWS Secrets Manager secret per router named routers/<endpoint>, held in the network account (210987654321) and synced cross-account by External Secrets, mounted at /etc/router-fleet/credentials. Each becomes a file named for the bare endpoint holding {"username","password"[,"port","use_ssl"]}.

The router-fleet-credentials Secret

ESO finds Secrets Manager secrets by the ^routers/ prefix via the aws-secrets-manager-network ClusterSecretStore (backed by the router-fleet-secret-reader role) and rewrites routers/(.*) → $1, so each router lands as /etc/router-fleet/credentials/<endpoint>. Adding a login is a one-secret operation in network; do not set a Description — the resolver keeps it in sync with the inventory site. See the memory note router-fleet-credentials secret for the curation history (77 routers, shared curator client credential). Manifest: externalsecret.yaml.

router-fleet-resolver

What it does

Source: src/router-fleet-resolver/main.py. Every hour the resolver:

  1. Loads the inventory (routers-inventory.yml; accepts flat or grouped shapes, YAML or JSON) and optionally augments it with AWS Site-to-Site VPN discovery — enumerating customer-gateway outside IPs via ec2:DescribeVpnConnections / ec2:DescribeCustomerGateways (best-effort, gated by AWS_VPN_DISCOVERY, default off).
  2. Loads credentials from the per-router file directory (falls back to a legacy single creds.json blob or flat map). No creds at all ⇒ every router is blackbox-only.
  3. Probes each router that has a credential: a single authenticated RouterOS-API call (/system/identity/print) over api-ssl:8729, then plain api:8728. Success ⇒ the router joins the mktxp (authenticated) tier with the working transport; failure ⇒ it stays blackbox-only and the failure is classified into a stable auth_status (invalid_credentials, timeout, connection_refused, dns_error, ssl_error, unreachable, error).
  4. Publishes the two configs via the Kubernetes API (see below) and rolls the mktxp Deployment by patching a router-fleet-resolver/config-hash annotation (no stakater/reloader in-cluster; an unchanged config hash is a no-op roll).
  5. Reconciles secret Descriptions and self-annotates the inventory in git (best-effort).

Outputs

Output Kind Consumed by
mktxp-config Secret (mktxp.conf) — one [section] per authenticated router, all mktxp collectors enabled (health, interface, ipsec, poe, wireless, dhcp, …) the mktxp Deployment in observability
router-blackbox-icmp Prometheus-Operator ScrapeConfig (monitoring.coreos.com/v1alpha1) — one staticConfig per inventory endpoint carrying site_name / router_endpoint / client / tier / auth_status labels; /probe against blackbox-exporter.observability.svc:9115 (module icmp_camera, reused) Prometheus

Section naming and labels

The mktxp section name is a slug of the site (_slug()), so mktxp's per-router metric label maps cleanly to site_name via relabeling. The blackbox ScrapeConfig's tier label is authenticated when mktxp also covers the router, else blackbox; auth_status records why (surfaced as the "Auth" column on the Routers dashboard).

Self-annotating inventory (git write-back)

Like camera-probe-propagator, the resolver commits the inventory .yml back to platform-utils via the platform-bot GitHub App (GITHUB_APP_ID: "1234567", private key at /secrets/github/private-key.pem). It:

  • sets an inline # authenticated / # blackbox comment on each - endpoint: line (whether a routers/<endpoint> secret exists); and
  • files any orphaned secret (a routers/* with no inventory entry) as a real entry under a - client: orphaned group, and prefixes that secret's SM Description with ORPHANED - (dropped once you move the entry to a real client).

Secret-Description reconciliation and orphan filing assume the cross-account NETWORK_SECRET_ROLE (arn:aws:iam::210987654321:role/router-fleet-resolver, in network) using a regional STS endpoint (af-south-1 is opt-in — global-endpoint tokens are rejected). All git/AWS steps are best-effort and idempotent.

Deployment

CronJob k8s/base/router-fleet-resolver/cronjob.yaml:

  • Schedule 17 * * * *; concurrencyPolicy: Forbid, restartPolicy: Never, backoffLimit: 1, activeDeadlineSeconds: 600, ttlSecondsAfterFinished: 86400.
  • Image ghcr.io/example-org/router-fleet-resolver:latest (Flux-managed tag in the prod overlay), pulled with imagePullSecrets: router-fleet-resolver-ghcr.
  • Resources requests 25m / 64Mi, limits 250m / 192Mi; hardened securityContext (runAsNonRoot, UID 1000, readOnlyRootFilesystem, drop ALL caps, RuntimeDefault seccomp).
  • Volumes: inventory (ConfigMap router-fleet-inventory), credentials (Secret router-fleet-credentials, optional), github-key (Secret router-fleet-resolver-github), and an emptyDir /tmp.

Env (selected): NAMESPACE=observability, MKTXP_DEPLOYMENT=mktxp, MKTXP_SECRET=mktxp-config, AWS_VPN_DISCOVERY=false, AWS_REGION=af-south-1, AWS_STS_REGIONAL_ENDPOINTS=regional, NETWORK_SECRET_ROLE=…:role/router-fleet-resolver, GITHUB_APP_ID=1234567, INVENTORY_REPO_PATH=k8s/base/router-fleet-resolver/routers-inventory.yml.

This is the one job here that needs the ServiceAccount token

Unlike the sibling exporters, the resolver's whole purpose is to call the Kubernetes API, so it mounts its SA token. RBAC (rbac.yaml) is a namespace-scoped, least-privilege Role: get/create/update/patch on secrets; get/list/create/update/patch on scrapeconfigs; get/patch on deployments (to roll mktxp). No cluster-wide access, no delete, no wildcards. Cross-account AWS is via EKS Pod Identity on the same ServiceAccount.

Config / secrets it needs

  • ConfigMap router-fleet-inventory (the inventory).
  • Secret router-fleet-credentials (per-router logins from network).
  • Secret router-fleet-resolver-ghcr (ghcr image pull, from SM github/image-pull).
  • Secret router-fleet-resolver-github (platform-bot App key github/platform-bot-private-key.pem).
  • IAM: Pod Identity role granting sts:AssumeRole on NETWORK_SECRET_ROLE plus (if enabled) EC2 VPN describe for discovery.

Image / language

Python 3.12-slim (Dockerfile, non-root appuser UID 1000, no HEALTHCHECK — one-shot job). Deps (requirements.txt): librouteros==3.4.1, kubernetes==31.0.0, boto3==1.35.99, PyYAML==6.0.2, requests==2.32.3, PyJWT==2.10.1, cryptography==44.0.0.

router-lifetime-reconciler

What it does

Source: src/router-lifetime-reconciler/main.py. This is the counterpart to the terraform/aws/_modules/vpn rekey settings: it keeps the MikroTik (customer) side of each AWS Site-to-Site VPN tunnel's IPsec lifetimes and DPD aligned with the AWS side, so a wedged tunnel self-heals (DPD tears down the dead peer and renegotiates) instead of needing a manual disable/enable on the router — the failure mode that took Client A/Client C/Client B down for ~3 days on 2026-07-17.

Targets (match the vpn Terraform module and docs/runbooks/vpn-mikrotik-ipsec.md):

Object Field Target (env)
/ip ipsec profile lifetime 8h (PROFILE_LIFETIME)
/ip ipsec profile dpd-interval / dpd-maximum-failures 10s / 3 (DPD_INTERVAL / DPD_MAX_FAILURES)
/ip ipsec proposal lifetime 1h (PROPOSAL_LIFETIME)

For each router it can authenticate to, it connects (reusing the resolver's api-ssl→api pattern), reads /ip/ipsec/peer|profile|proposal|policy, identifies the AWS peer (by matching the peer address against AWS_TUNNEL_IPS when provided, else the PEER_MATCH regex, default (?i)aws), computes the drift of the referenced profile / proposal from target, and either logs it (audit) or sets it (apply). RouterOS time values are normalised to seconds for idempotent comparison. Other IPsec (WireGuard-adjacent, non-AWS site-to-site) is never touched.

Audit-first (like the tunnel-trampoline dry-run soak)

  • APPLY=false (default): READ-ONLY. Connects with the existing monitoring credentials and logs the drift only. Nothing is written to any router. This is the shipped state.
  • APPLY=true: writes. Uses a separate write-capable credential at /etc/router-admin/credentials/<endpoint> (from routers-admin/<endpoint> secrets), and only sets a field whose current value differs from target (idempotent). A router with no write credential is audited read-only and skipped for writes. The reconciler never fails the CronJob on drift (returns 0) — drift is expected until APPLY is enabled.

Enabling writes (deliberate, per the runbook)

  1. Provision a write-capable RouterOS user and store it as routers-admin/<endpoint> in network Secrets Manager. 2. Grant the fleet secret-reader GetSecretValue on routers-admin/* and add an ExternalSecret syncing them to /etc/router-admin/credentials (the router-admin-credentials Secret is not synced yet). 3. Validate on one canary (Client A) with APPLY=true scoped to it, confirm /ip ipsec active-peers re-establishes cleanly, then widen.

Deployment

CronJob k8s/base/router-lifetime-reconciler/cronjob.yaml:

  • Schedule 37 * * * * (offset from the resolver's :17 so they don't hit the routers together); same job-control settings as the resolver.
  • Image ghcr.io/example-org/router-lifetime-reconciler:latest (Flux-managed tag), pull secret router-lifetime-reconciler-ghcr.
  • automountServiceAccountToken: false — audit mode needs no Kubernetes API and no AWS; it only reads mounted files and talks to routers. (Attach a Pod Identity role later only if API-based AWS_TUNNEL_IPS correlation is added.)
  • Resources requests 25m / 64Mi, limits 250m / 128Mi; same hardened securityContext as the resolver.
  • Volumes: inventory and credentials reused by name from router-fleet-resolver (ConfigMap router-fleet-inventory, Secret router-fleet-credentials), plus admin-credentials (Secret router-admin-credentials, optional, for APPLY) and an emptyDir /tmp.

Env (shipped): APPLY=false, PROFILE_LIFETIME=8h, PROPOSAL_LIFETIME=1h, DPD_INTERVAL=10s, DPD_MAX_FAILURES=3. Also honoured: PEER_MATCH, AWS_TUNNEL_IPS (JSON {endpoint: [ip,…]}), WRITE_CREDENTIALS_PATH, PROBE_TIMEOUT (default 8).

Config / secrets it needs

  • ConfigMap router-fleet-inventory + Secret router-fleet-credentials (both owned by the resolver).
  • Secret router-lifetime-reconciler-ghcr (ghcr image pull).
  • Secret router-admin-credentials (write creds) — only when APPLY=true; not synced yet.

Image / language

Python 3.12-slim (Dockerfile, non-root UID 1000). Deps (requirements.txt): librouteros==3.4.1, PyYAML==6.0.2 only. python test_reconciler.py covers the pure logic (RouterOS time parsing, drift detection, AWS-peer identification).

How platform-infra consumes these

platform-infra does not hold the manifests — it points Flux at the manifests in platform-utils and drives image automation:

  • Flux Kustomizations (kubernetes/apps/utils/overlays/prod/router-fleet-resolver/flux-kustomization.yaml and the reconciler's) target path: ./k8s/overlays/prod/<component> inside the platform-utils GitRepository, prune: true, wait: true, 5-minute interval.
  • Image automation — an ImageRepository (base) scans ghcr.io/example-org/<component> (pull secret ghcr-credentials), an ImagePolicy selects semver tags ^v[0-9]+\.[0-9]+\.[0-9]+$ (>=1.0.0), and an ImageUpdateAutomation writes the chosen tag into the platform-utils prod overlay's newTag setter (# {"$imagepolicy": "flux-system:<component>:tag"} in k8s/overlays/prod/router-fleet-resolver/kustomization.yaml). So a new image built by the platform-utils docker-bake targets (router-fleet-resolver / router-lifetime-reconciler) rolls out automatically.

Observability wiring (the consumer of the resolver's output)

The resolver's outputs feed the prod observability stack, which alerts on them (prometheusrule-router-icmp.yaml, prometheusrule-router-health.yaml):

  • The router-blackbox-icmp ScrapeConfig drives ClientRouterDown, RouterReachabilityDegraded, RouterHighLatency, RouterBlackboxProbesMissing (all on probe_*{job="router-blackbox-icmp"}, labelled by site_name / router_endpoint / tier).
  • The mktxp-config Secret drives mktxp, whose mktxp_* metrics power RouterHighCPU, RouterHighMemory, RouterHighTemperature, RouterDiskFull, RouterInterfaceErrors, RouterUnexpectedReboot, RouterLowVoltage, and MktxpExporterDown.
  • A RouterFleetResolverStalled alert fires when kube_cronjob_status_last_schedule_time{cronjob="router-fleet-resolver"} is older than 2h — i.e. the tiering config and blackbox target list have gone stale.