¶ Storage Architecture
See full diagram gallery for interactive versions
The cluster uses two storage layers: Longhorn for persistent block volumes (RWO) and NFS for shared media and monitoring data (RWX/ROX). All storage runs on the same hardware, backed by Proxmox VM disks (Longhorn) and the Ugreen DXP4800 NAS (NFS).
¶ Storage Layers Overview
¶ StorageClasses
| StorageClass | Provisioner | Access Mode | Encryption | Used By |
|---|---|---|---|---|
longhorn |
driver.longhorn.io |
RWO | No | Most app PVCs — databases, configs |
longhorn-encrypted |
driver.longhorn.io |
RWO | LUKS2 AES-256-XTS | Open WebUI (chat history, uploads) |
nfs-monitoring |
NFS static | RWX | No | Prometheus, Loki, AlertManager |
nfs-gitea |
NFS static | RWO | No | Gitea package registry (20Gi) |
nfs-rag |
NFS static | RWX | No | Qdrant vector store + ingester |
| (empty string) | Static binding | RWX/ROX | No | Media namespace direct NFS PVs |
longhornis the default StorageClass for all dynamically-provisioned PVCs. IfstorageClassNameis omitted, Longhorn claims the volume.
¶ Longhorn Node Distribution
agent-4 excluded from Longhorn: The NVMe on pve4 is aging and not scheduled for Longhorn replicas. GPU workloads on agent-4 use
emptyDiror NFS, never Longhorn PVCs.
¶ Encrypted Volumes
The longhorn-encrypted StorageClass uses Longhorn's built-in LUKS2 encryption:
- Encryption key:
longhorn-cryptoSecret inlonghorn-systemnamespace - Critical: If
longhorn-cryptois deleted, all encrypted volumes become unrecoverable - Currently used by: Open WebUI (chat history, user preferences)
- Manifest:
kubernetes/apps/open-webui/longhorn-encrypted.yaml
Recovery warning: The
longhorn-cryptoSecret must be backed up separately (e.g., exported to a password manager). Velero does NOT back up Secrets inlonghorn-systemby default. Without the key, encrypted PVCs cannot be decrypted.
Bootstrap the encryption secret:
kubectl create secret generic longhorn-crypto \
--from-literal=CRYPTO_KEY_VALUE=<strong-random-key> \
--from-literal=CRYPTO_KEY_PROVIDER=secret \
-n longhorn-system
¶ NFS Mounts — Media Namespace
The media namespace uses static PVs bound directly to NFS exports on the DXP4800. These use storageClassName: "" (no dynamic provisioner — manual PV/PVC binding).
¶ NAS Service Accounts
All NAS accounts are members of media-services group (GID 10000). The NFS export uses root_squash — pods must run as the matching UID or GID to access files.
| NAS Account | UID | Access Scope | Service |
|---|---|---|---|
| svc-jellyfin | 10010 | Read all media | Jellyfin |
| svc-bazarr | 10011 | Read movies/ + tv/ | Bazarr |
| svc-rclone | 10012 | Read-write all (also used by qBittorrent temporarily) | Seedbox Sync, qBittorrent |
| svc-radarr | 10013 | Read-write movies/ + downloads/movies/ | Radarr |
| svc-sonarr | 10014 | Read-write tv/ + downloads/tv/ | Sonarr |
| svc-prowlarr | 10015 | Config only (no NFS) | Prowlarr |
| svc-tdarr | 10016 | Read-write all (transcode-in-place) | Tdarr |
| svc-media-ctrl | 10017 | Read-write auto-managed/ + permanent/ | Media Controller |
| svc-plex | 10018 | Read all media | Plex |
TODO: Create
svc-qbittorrent(UID 10019) on NAS for proper isolation. qBittorrent currently runs assvc-rclone(UID 10012), which has broader permissions than needed.
¶ NFS PV Map
All media PV definitions: kubernetes/apps/media/nfs-pv.yaml
¶ NAS Directory Layout
/volume1/media/
movies/ (mode 2775, group: media-services)
tv/ (mode 2775)
music/ (mode 2755)
downloads/
movies/ (staging for Radarr import)
tv/ (staging for Sonarr import)
staging/ (rsync destination from seedbox — Radarr/Sonarr scan here)
auto-managed/ (Media Controller tier — auto-delete after threshold)
movies/
tv/
permanent/ (Media Controller tier — keep forever)
movies/
tv/
jellyfin-config/ (Jellyfin config shared across HA replicas)
jellyfin-transcode/ (transcode scratch, shared)
¶ NFS Mounts — Other Namespaces
| Namespace | Mount | StorageClass | Size | Purpose |
|---|---|---|---|---|
| monitoring | Prometheus TSDB | nfs-monitoring | 30Gi | Prometheus time-series data (14d retention) |
| monitoring | Loki data | nfs-monitoring | 10Gi | Log storage (72h retention) |
| monitoring | AlertManager data | nfs-monitoring | 1Gi | Alert state |
| gitea | Gitea data | nfs-gitea | 20Gi | Package registry (PyPI, npm, Go, generic) |
| arc-runner-system | actions-cache | static PV | 50Gi | GitHub Actions cache (NAS /volume1/actions-cache) |
| rag | Qdrant vector store | nfs-rag | dynamic | Embedding vectors |
| rag | rag-ingester staging | nfs-rag | dynamic | Document drop zone |
¶ PVC Allocation by Namespace
Longhorn PVCs (dynamically provisioned, RWO):
| Namespace | PVC | Size | Purpose |
|---|---|---|---|
| authentik | authentik-postgres-data | 5Gi | Authentik PostgreSQL |
| media | jellyfin-postgres | 5Gi | Jellyfin PostgreSQL |
| media | plex-config | 10Gi | Plex metadata, database, plugins |
| media | media-controller-postgres | 5Gi | Media controller PostgreSQL |
| media | bazarr-config | 1Gi | Bazarr config |
| media | jellyseerr-config | 2Gi | Jellyseerr config |
| media | prowlarr-config | 1Gi | Prowlarr config |
| media | sonarr-config | 2Gi | Sonarr config |
| media | radarr-config | 2Gi | Radarr config |
| media | tdarr-server-config | 5Gi | Tdarr server data |
| media | qbittorrent-config | 2Gi | qBittorrent config |
| open-webui | open-webui-data | 5Gi | Open WebUI data (longhorn-encrypted) |
| wiki | postgres-data | 2Gi | Wiki.js PostgreSQL |
| cardboard | cardboard-postgres | 5Gi | Cardboard TCG PostgreSQL |
| trade-bot | trade-bot-postgres | 2Gi | Trade Bot PostgreSQL |
| ham | ham-postgres | 5Gi | HAM PostgreSQL |
| digital-signage | ds-postgres | 5Gi | Digital Signage PostgreSQL |
| aja-recipes | recipes-postgres | 10Gi | Recipes PostgreSQL |
| media-profiler | media-profiler-postgres | 5Gi | Media Profiler PostgreSQL |
| dnd | dnd-postgres | 10Gi | D&D Platform PostgreSQL + pgvector |
| auto-brand | auto-brand-postgres | 5Gi | Auto Brand PostgreSQL |
| jupyter | jupyter-data | 5Gi | Jupyter notebooks |
| rag | qdrant-storage | dynamic | Qdrant vector storage (nfs-rag) |
¶ Namespace Storage Classification
At-a-glance view of which storage backend each namespace depends on.
media namespace uses both: Longhorn PVCs for postgres + application config, and static NFS PVs for the 10TB media library. These are distinct — app state on Longhorn (replicated, RWO), media files on NFS (shared RWX/ROX across all arr services).
¶ Backup Pipeline
Longhorn backup credentials:
longhorn-s3-credentialsSecret inlonghorn-systemnamespace. Generated fromterraform/environments/awsoutputs — never committed to git.
Known Issue — etcd-backup image tag:
harbor.../production/etcd-backup:latestuses:latest(policy violation — production images should use SHA tags). The image hasimagePullPolicy: Alwaysas a workaround. Track: rungh workflow run promote-image.ymlwhen a SHA tag is cut.
All 15 PostgreSQL/TimescaleDB databases have automated daily S3 backup CronJobs (3:00–4:05 AM UTC). The full schedule is shown above.
TimescaleDB note:
polymarket-labbackup uses standardpg_dump— compatible but restores hypertables as regular tables. Future improvement: migrate totimescaledb-backupfor full chunk-level recovery.
auto-brand not yet backed up: PostgreSQL, NATS, and Redis StatefulSets in the
auto-brandnamespace have no backup CronJob. Helm chart manages the deployment — backup CronJob needs to be added tokubernetes/apps/auto-brand/.
¶ Manifests
kubernetes/core/etcd-backup/cronjob.yaml -- etcd S3-uploader CronJob (kube-system, control-plane nodeSelector)
kubernetes/core/longhorn-recurring-jobs.yaml -- daily-snapshot + daily-s3-backup RecurringJobs
kubernetes/core/longhorn-s3-config.yaml -- S3 backup target + credential Secret template
kubernetes/apps/open-webui/longhorn-encrypted.yaml -- longhorn-encrypted StorageClass definition
kubernetes/apps/media/nfs-pv.yaml -- All static NFS PVs for media namespace
kubernetes/apps/media/jellyfin-config-nfs.yaml -- Jellyfin NFS config PVs
kubernetes/apps/media/jellyfin-transcode-rwx.yaml -- Jellyfin transcode NFS PV
¶ Related Pages
- Velero — Kubernetes object backup: schedules, S3 bucket, restore procedures
- NAS Exporter — NFS health monitoring, volume usage metrics, Grafana dashboard
- Observability Stack — Prometheus NFS storage (nfs-monitoring), AlertManager, Loki
- Cluster Architecture Overview — Full diagram catalog (includes network topology and namespace layout)
- Media Stack — NAS service accounts, directory layout, NFS PV details