¶ Ingress Architecture
See full diagram gallery for interactive versions
The cluster uses two distinct domains with different access controls:
*.k3s.strommen.systems— public internet-facing, all routes require authentication*.k3s.internal.strommen.systems— internal only, resolved via Route53 to private IPs, LAN/VPN access only
All traffic enters through a single load-balancer IP (192.168.20.200, assigned by MetalLB) and is routed by Traefik.
¶ Network Path
¶ DDNS — Dynamic Public IP
The cluster has a residential/dynamic public IP. A CronJob in the public-ingress namespace keeps Route53 updated:
| Detail | Value |
|---|---|
| Managed record | origin.k3s.strommen.systems (A record, TTL 300) |
| Hosted zone | Z1LLBOMFGEFI6S |
| Schedule | Every 5 minutes |
| Mechanism | CronJob (ddns-updater) — checks public IP via ifconfig.me / ipify / fallbacks, calls aws route53 change-resource-record-sets only on change |
| IAM | Shares cert-manager IAM user (already has Route53 write permissions) |
| Credentials secret | ddns-aws-credentials in public-ingress namespace |
| Manifest | kubernetes/apps/public-ingress/ddns-cronjob.yaml |
CloudFront distributes *.k3s.strommen.systems as an alias record (managed by Terraform). It connects to origin.k3s.strommen.systems as its origin — the DDNS CronJob keeps that origin record pointing at the current public IP.
¶ TLS Certificates
| Certificate | Namespace | Covers | Issuer |
|---|---|---|---|
public-wildcard-tls |
public-ingress |
*.k3s.strommen.systems |
letsencrypt-prod (DNS-01) |
| Per-service certs | various | *.k3s.internal.strommen.systems |
letsencrypt-prod (DNS-01) |
All certificates managed by cert-manager with Route53 DNS-01 challenge.
¶ Public Routes (*.k3s.strommen.systems)
All public IngressRoutes live in the public-ingress namespace and use public-wildcard-tls.
¶ Authentik forwardAuth Protected
Users must authenticate with Google via Authentik. Session cookie valid across all *.k3s.strommen.systems apps.
| URL | Backend | Namespace | IngressRoute source |
|---|---|---|---|
| chat.k3s.strommen.systems | open-webui:80 | open-webui | public-ingress.yaml |
| dash.k3s.strommen.systems | cluster-dashboard:80 | default | public-ingress.yaml |
| auto-brand.k3s.strommen.systems | auto-brand-web-ui:80 | auto-brand | public-ingress.yaml |
| ham.k3s.strommen.systems | ham-web:80 | ham | public-ingress.yaml |
| jellyseerr.k3s.strommen.systems | jellyseerr:5055 | media | public-ingress.yaml |
| bazarr.k3s.strommen.systems | bazarr:6767 | media | public-ingress.yaml |
| radarr.k3s.strommen.systems | radarr | media | media/radarr.yaml |
| sonarr.k3s.strommen.systems | sonarr | media | media/sonarr.yaml |
| prowlarr.k3s.strommen.systems | prowlarr | media | media/prowlarr.yaml |
| tdarr.k3s.strommen.systems | tdarr | media | media/tdarr.yaml |
| media-controller.k3s.strommen.systems | media-controller:8080 | media | media/media-controller.yaml |
| store.k3s.strommen.systems | storefront:80 | tshirt-cannon | public-ingress.yaml |
¶ Self-Auth (No forwardAuth Middleware)
These services expose public endpoints but handle their own authentication:
| URL | Backend | Auth method | Notes |
|---|---|---|---|
| ha.k3s.strommen.systems | home-assistant.home-assistant:8123 | HA native login | Google Assistant webhook also uses this domain |
| jellyfin.k3s.strommen.systems | jellyfin.media | Jellyfin native + OIDC plugin | Web uses Authentik SSO via jellyfin-plugin-sso; TV/mobile apps use native auth |
| plex.k3s.strommen.systems | plex.media | Plex account auth | External clients authenticate directly to Plex |
HA webhook exception:
ham.k3s.strommen.systems/api/strava/webhookhas a separate IngressRoute (priority 200) that bypasses Authentik — Strava push subscription verification calls this path directly.
¶ Open Gmail OAuth2 Proxy
Media Profiler uses a separate oauth2-proxy-open-gmail instance (reverse proxy mode) that allows any @gmail.com account. Login events are logged to stdout (queryable in Grafana via Loki).
| URL | Auth | Backend |
|---|---|---|
| media-profiler.k3s.strommen.systems | Any @gmail.com | oauth2-proxy-open-gmail → media-profiler.media-profiler:80 |
¶ Open Public (No Authentication)
These routes have no auth middleware — fully public, no login required.
| URL | Backend | Namespace | Notes |
|---|---|---|---|
| steve-lee.k3s.strommen.systems | steve-lee:80 | open-webui | Public ceramics portfolio — no auth required |
¶ OAuth2 Proxy Callback (Legacy — Soak Period)
The original OAuth2 Proxy is still deployed for rollback safety. Scheduled for removal on 2026-04-11 after 7-day soak period.
| URL | Purpose |
|---|---|
| auth.k3s.strommen.systems/oauth2/ | OAuth2 Proxy callback route (deprecated, kept for rollback) |
¶ Internal Routes (*.k3s.internal.strommen.systems)
Internal hostnames resolve to private IPs via Route53 (RFC 1918 — low risk). Accessible from LAN or WireGuard VPN only. No forwardAuth middleware by default — services use native auth or rely on network-level access control.
¶ Infrastructure & Tooling
| URL | Backend | Namespace | Auth |
|---|---|---|---|
| auth.k3s.internal.strommen.systems | authentik-server:9000 | authentik | Authentik admin UI |
| harbor.k3s.internal.strommen.systems | harbor | harbor | Authentik OIDC + local accounts |
| gitea.k3s.internal.strommen.systems | gitea:3000 | gitea | Authentik OIDC (no local registration) |
| grafana.k3s.internal.strommen.systems | grafana | monitoring | Authentik OIDC + local admin |
| prometheus.k3s.internal.strommen.systems | prometheus-kube-prometheus-prometheus:9090 | monitoring | None (internal trust) |
| alertmanager.k3s.internal.strommen.systems | prometheus-kube-prometheus-alertmanager:9093 | monitoring | None (internal trust) |
| longhorn.k3s.internal.strommen.systems | longhorn-frontend.longhorn-system:80 | longhorn-system | None (internal trust) |
| wiki.k3s.internal.strommen.systems | wiki:80 | wiki | Authentik OIDC |
| gha.k3s.internal.strommen.systems | gha-dashboard | gha-dashboard | None (internal trust) |
| aws-lens.k3s.internal.strommen.systems | aws-lens:3000 | aws-lens | None (internal trust) |
| jupyter.k3s.internal.strommen.systems | jupyter | jupyter | Token-based (jupyter-token secret) |
| qdrant.k3s.internal.strommen.systems | qdrant | rag | API key |
Prometheus, Alertmanager, and Longhorn internal Ingress routes are defined in
kubernetes/apps/cluster-dashboard/service-ingresses.yaml.
¶ Applications
| URL | Backend | Namespace | Auth |
|---|---|---|---|
| chat.k3s.internal.strommen.systems | open-webui:80 | open-webui | Authentik OIDC (app-native) |
| openclaw.k3s.internal.strommen.systems | openclaw:80 | open-webui | Authentik OIDC (openclaw gateway) |
| ha.k3s.internal.strommen.systems | home-assistant:8123 | home-assistant | HA native login |
| ham.k3s.internal.strommen.systems | ham-web:80 | ham | None (internal trust) |
| recipes.k3s.internal.strommen.systems | aja-recipes | aja-recipes | None |
| cardboard.k3s.internal.strommen.systems | cardboard:80 | cardboard | Authentik proxy (owner-only policy: mstrommen@gmail.com) |
| ds.k3s.internal.strommen.systems | digital-signage | digital-signage | Authentik forwardAuth (Pi kiosk displays use MQTT direct bypass) |
| media-profiler.k3s.internal.strommen.systems | media-profiler | media-profiler | None (internal) |
| qbt.k3s.internal.strommen.systems | qbittorrent | media | qBittorrent native |
Cardboard Authentik proxy: Configured via Terraform (
terraform/environments/google/cardboard-access.tf) usingauthentik_provider_proxyinforward_singlemode. Expression policy restricts access tomstrommen@gmail.comonly. This is an embedded outpost proxy (different from forwardAuth middleware — the Authentik outpost intercepts the request rather than a Traefik middleware forwarding it).
Digital Signage auth note: The Ingress has
public-ingress-authentik-forward-auth@kubernetescrdmiddleware — browser access requires Authentik login. Pi kiosk displays communicate via MQTT broker directly (not through the web UI), so they are unaffected by Authentik.
¶ Media Management (Internal)
| URL | Backend | Namespace | Notes |
|---|---|---|---|
| (public Authentik) | radarr | media | Public URL also exists |
| (public Authentik) | sonarr | media | Public URL also exists |
| (public Authentik) | prowlarr | media | Public URL also exists |
| (public Authentik) | bazarr | media | Public URL also exists |
¶ D&D Platform
| URL | Backend |
|---|---|
| dnd.k3s.internal.strommen.systems | dnd-frontend.dnd:80 |
| api.dnd.k3s.internal.strommen.systems | dnd-backend.dnd |
| voice.dnd.k3s.internal.strommen.systems | livekit.dnd |
¶ Middleware Reference
| Middleware | Namespace | Type | Purpose |
|---|---|---|---|
authentik-forward-auth |
public-ingress |
ForwardAuth | Authentik proxy outpost — validates session, redirects to Google if unauthed |
google-oauth |
public-ingress |
ForwardAuth | DEPRECATED OAuth2 Proxy (kept for rollback until 2026-04-11) |
oauth2-redirect-errors |
public-ingress |
Errors | Catches 401 from forwardAuth, redirects browser to /oauth2/start?rd={url} |
wiki-security-headers |
wiki |
Headers | HSTS, CSP, XSS, framing headers for Wiki.js |
¶ auth.k3s.strommen.systems vs auth.k3s.internal.strommen.systems
Both domains point to the same Authentik server. The split is for network-layer segregation:
- External users authenticate via
auth.k3s.strommen.systems(goes through UDM Pro port forward) - Internal services call the Authentik API via cluster DNS (
authentik-server.authentik.svc.cluster.local) - The
auth.k3s.internal.strommen.systemsIngressRoute provides a human-readable URL for LAN/VPN admin access
¶ Known Cleanup Items
| Priority | Item | Location | Action |
|---|---|---|---|
| ✅ Fixed | media-controller public API unauthenticated |
kubernetes/apps/media/media-controller.yaml |
Fixed 2026-04-05 — authentik-forward-auth middleware added |
| ✅ Fixed | Orphaned jellyseerr-tls Certificate |
kubernetes/apps/media/jellyseerr.yaml |
Removed 2026-04-05 |
| ✅ Fixed | Orphaned bazarr-tls Certificate |
kubernetes/apps/media/bazarr.yaml |
Removed 2026-04-05 |
| ✅ Fixed | OpenClaw public route not manifested | kubernetes/apps/public-ingress/public-ingress.yaml |
Fixed 2026-04-05 — openclaw-chat-public IngressRoute added |
| ✅ Fixed | Duplicate IngressRoute for steve-lee | kubernetes/apps/steve-lee/public-ingress/ |
Removed 2026-04-05 — orphaned public-ingress/ subdirectory deleted |
| 🟡 Medium | Deprecated google-oauth middleware |
kubernetes/apps/public-ingress/public-ingress.yaml |
Remove after 2026-04-11 soak period — delete OAuth2 Proxy Deployment + google-oauth Middleware |
¶ Manifests
kubernetes/apps/public-ingress/public-ingress.yaml -- All public-ingress namespace resources:
namespace, wildcard cert, oauth2-proxy (both),
middlewares, all public IngressRoutes for
chat/dash/auto-brand/ha/jellyseerr/bazarr/store/ham/media-profiler
kubernetes/apps/public-ingress/ddns-cronjob.yaml -- DDNS updater CronJob (origin.k3s.strommen.systems)
kubernetes/apps/cluster-dashboard/service-ingresses.yaml -- Internal routes for Prometheus, Alertmanager, Longhorn
kubernetes/apps/media/radarr.yaml -- radarr public IngressRoute + cert
kubernetes/apps/media/sonarr.yaml -- sonarr public IngressRoute + cert
kubernetes/apps/media/prowlarr.yaml -- prowlarr public IngressRoute + cert
kubernetes/apps/media/tdarr.yaml -- tdarr public IngressRoute + cert
kubernetes/apps/media/media-controller.yaml -- media-controller IngressRoute + cert + authentik-forward-auth
kubernetes/apps/media/jellyfin.yaml -- jellyfin public IngressRoute + cert
kubernetes/apps/media/plex.yaml -- plex public IngressRoute + cert
kubernetes/apps/home-assistant/ingress.yaml -- ha internal IngressRoute + cert
terraform/environments/google/cardboard-access.tf -- Authentik proxy provider + policy for cardboard (owner-only)