FigJam Diagram: Cardboard — TCG Price Tracker Pipeline (expires 2026-04-13)
Tracks trading card game prices across 10 TCGs with automated scraping, historical charts, and portfolio P/L analysis.
cardboardmstrommen@gmail.com)harbor.k3s.internal.strommen.systems/production/cardboardPokemon, Magic: The Gathering, Yu-Gi-Oh!, Lorcana, One Piece, Weiss Schwarz, Flesh and Blood, Star Wars Unlimited, Digimon, Vanguard
| Resource | Type | Image | Notes |
|---|---|---|---|
cardboard |
Deployment (HPA: 1–3) | harbor.../production/cardboard:<sha> |
Flask :5555; /metrics, /api/stats |
cardboard-scraper |
CronJob — daily 23:00 UTC | harbor.../production/cardboard-scraper:<sha> |
Playwright + Chromium; 15-min timeout; backoffLimit: 2 |
cardboard-set-scraper |
CronJob — Mondays 02:00 UTC | harbor.../production/cardboard-scraper:<sha> |
Sealed products (--discover --limit-sets 10 --limit-cards 200); 1hr timeout |
postgres |
StatefulSet | postgres:16-alpine |
2Gi Longhorn PVC; daily S3 backup 3:00 AM UTC |
The webapp holds a cardboard-webapp ServiceAccount with the scraper-job-manager Role, allowing it to create and manage scraper Jobs on demand (user-triggered scrapes from the UI).
| Component | CPU Request | CPU Limit | Memory Request | Memory Limit |
|---|---|---|---|---|
| cardboard webapp | 50m | 500m | 128Mi | 256Mi |
| PostgreSQL | 50m | 500m | 128Mi | 512Mi |
| cardboard-scraper (daily) | 100m | 1 core | 512Mi | 1Gi |
| cardboard-set-scraper (weekly) | 200m | 2 cores | 512Mi | 2Gi |
Scrapers need high memory limits for Playwright + Chromium headless browser processes. The set-scraper runs longer with more pages so gets a higher CPU limit.
| Setting | Value |
|---|---|
| Min replicas | 1 |
| Max replicas | 3 |
| Scale up trigger | CPU or memory >70% |
| Scale down | 5-min stabilization window; max 50% per minute |
| Scale up | Immediate (0s stabilization); max 2 pods per 30s |
TCGPlayer (Set-Level, API-based)
TCGPlayer (Individual Items, Playwright)
eBay (Sold Listings)
| Method | Path | Description |
|---|---|---|
| GET | /api/items |
All tracked items |
| GET | /api/price-history/<name> |
Price history for an item |
| GET | /api/changes?days=N |
Price movements over N days |
| GET | /api/profit-loss |
Portfolio P/L with CAGR |
| GET | /api/games |
All supported card games |
| GET | /api/games/<code>/sets |
Sets for a specific game |
| GET | /api/sets/<code>/<slug>/prices |
Latest card prices in a set |
| POST | /api/update-prices |
Trigger on-demand scraper Job |
| GET | /metrics |
Prometheus metrics |
PostgreSQL tables: items, price_history, card_games, card_sets, set_price_history
Credentials in postgres-credentials secret (namespace: cardboard) — managed out-of-band, not in Git.
PW="$(openssl rand -base64 24)"
kubectl create secret generic postgres-credentials -n cardboard \
--from-literal=POSTGRES_USER=cardboard \
--from-literal=POSTGRES_PASSWORD="${PW}" \
--from-literal=POSTGRES_DB=cardboard \
--from-literal=DATABASE_URL="postgresql://cardboard:${PW}@postgres:5432/cardboard"
| Secret | Key(s) | Used For |
|---|---|---|
postgres-credentials |
POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DB, DATABASE_URL |
PostgreSQL connection |
cardboard-secrets |
flask-secret-key |
Flask SECRET_KEY env var (session security) |
harbor-pull-secret |
.dockerconfigjson |
Pull images from Harbor registry |
Secrets managed out-of-band — never committed to git.
Owner-only access enforced via Authentik expression policy (mstrommen@gmail.com only). Defined in terraform/environments/google/cardboard-access.tf.
Resources created:
authentik_provider_proxy.cardboard — forward_single mode on cardboard.k3s.internal.strommen.systemsauthentik_application.cardboard — wraps the providerauthentik_policy_expression.cardboard_owner_only — email match expressionauthentik_policy_binding.cardboard_owner — binds policy to application at order 0CronJob runs daily at 3:00 AM UTC → s3://k3s-homelab-backups-855878721457/postgres-backups/cardboard/
ServiceMonitor scrapes cardboard at /metrics every 30s. Label: release: prometheus.
GitHub Actions → build → Harbor staging → Trivy scan → promote to Harbor production → deploy via kubectl rollout restart
Pull secret: harbor-pull-secret (static, no expiry). All images are Harbor-only.
kubernetes/apps/cardboard/
cardboard.yaml -- Namespace, PVC, StatefulSet (postgres), Services, Deployment,
Ingress, ServiceMonitor, RBAC (ServiceAccount/Role/RoleBinding),
CronJobs (daily scraper + weekly set-scraper)
hpa.yaml -- HorizontalPodAutoscaler (1-3 replicas, CPU+memory 70%)
pdb.yaml -- PodDisruptionBudget
postgres-backup-cronjob.yaml -- Daily S3 backup at 3AM UTC
terraform/environments/google/cardboard-access.tf -- Authentik proxy provider + owner-only policy