184 lines
15 KiB
Markdown
184 lines
15 KiB
Markdown
# Homelab Monitor — Project Memory
|
|
|
|
Unified health dashboard for the homelab (infra docs: `jhodgkin/homelab` repo, `docs/infrastructure.md`).
|
|
One login instead of logging into Proxmox, Zabbix, OMV, and every service separately.
|
|
|
|
## How to resume work in a new session
|
|
|
|
1. List open issues: `GET /api/v1/repos/jhodgkin/homelab-monitor/issues?state=open&limit=30` on
|
|
`git.jerodrigged.com` (same pattern as the main homelab backlog, see global `~/.claude/CLAUDE.md`).
|
|
Milestone `v1-dashboard` is the current focus; everything else is backlog.
|
|
2. Pick an issue, comment that you're starting it, do the work, comment what you did, close it.
|
|
3. Commit and push after every meaningful step — don't let work sit uncommitted.
|
|
|
|
## Architecture
|
|
|
|
- `apps/api` — Fastify + TypeScript. Polls **collectors** on an interval (`POLL_INTERVAL_SECONDS`,
|
|
default 30s), writes snapshots to SQLite (`apps/api/src/db`), serves `/api/*`.
|
|
- `apps/web` — React + Vite + TypeScript. Polls `/api/hosts` every 15s, renders host/container cards.
|
|
- `config/hosts.yaml` — declares *what* to monitor (no secrets, committed). Credentials live in `.env`
|
|
(gitignored) — see `.env.example` for the full list.
|
|
- Collectors implement the `Collector` interface (`apps/api/src/collectors/types.ts`) and are registered
|
|
in `apps/api/src/index.ts`. Adding a new data source = new collector + config entry, no other wiring.
|
|
- `ProxmoxCollector` (done): one API call to `pve` returns CPU/mem/disk **and PSI pressure**
|
|
(`pressurecpusome`, `pressurememoryfull`) for the host + every LXC. Covers ~22 of ~24 machines.
|
|
- `SshHostCollector` (done): covers `.180` (omv) and `.171` (ripper), the bare-metal boxes Proxmox
|
|
can't see. Dedicated SSH key with a **forced `command=`** in the remote `authorized_keys` — the
|
|
server always runs a fixed read-only script regardless of what's exec'd, so a leaked key is still
|
|
contained. See `docs/ssh-collector-key-setup.md`, including the gotcha that `diskPaths` in
|
|
`hosts.yaml` must be kept in sync **by hand** with the remote script's `DISK_<label>=` lines.
|
|
- No Zabbix collector — see "Zabbix: intentionally skipped" below.
|
|
- **Historical sparklines** (done): `/api/hosts` embeds the last ~40 samples per host (one windowed
|
|
query, `ROW_NUMBER() OVER (PARTITION BY host_id ...)`), rendered as small hand-rolled SVG sparklines
|
|
in each card — no charting library. Fixed a real bug while building this: the Proxmox **host node's**
|
|
own mem/disk had been `null` the whole time (`/nodes/{node}/status` nests `memory`/`rootfs` objects;
|
|
code assumed the flat shape the *LXC listing* endpoint uses). LXC entries were never affected.
|
|
- **Hot-reload** (done): `sshHosts`/`knownDevices` in `config/hosts.yaml` are re-read on the next poll
|
|
cycle (mtime check) — no container restart needed. Proxmox hosts already needed no config (auto-
|
|
discovered every poll). See `docs/hot-reload.md` — includes a real single-file-bind-mount inode bug
|
|
that silently broke this on first deploy, fixed by mounting `./config` as a directory.
|
|
- Auth: local (bcrypt + signed session cookie) is **always** available — it's not an exclusive mode
|
|
switch. OIDC via Authentik is an *additional* sign-in button, live when `OIDC_ENABLED=true`. See
|
|
`docs/oidc-setup.md`. Deliberately additive so OIDC config problems can never lock out the admin login.
|
|
Two real bugs found only once someone actually clicked the button (fixed, see `docs/oidc-setup.md` and
|
|
`e2e/README.md`): `OIDC_ISSUER_URL` pointed at Authentik's LAN IP, which got baked into the
|
|
browser-facing `authorization_endpoint`; and the callback hardcoded `http://` when reconstructing the
|
|
current URL (Fastify never sees HTTPS, it terminates upstream), sending the wrong scheme as
|
|
`redirect_uri` in the token exchange. Both are exactly the class of bug curl-based verification cannot
|
|
catch — real redirect chains and a real IdP login form need a real browser.
|
|
- **Sign-out ends the full Authentik session, not just the app session** (issue #19, done): RP-Initiated
|
|
Logout via `openid-client`'s `buildEndSessionUrl` (`apps/api/src/auth/oidc.ts`), full-page navigation
|
|
(not `fetch`) so Authentik's browser cookie actually clears. Two more bugs found only via full
|
|
click-through testing, same pattern as above: the provider's `invalidation_flow` was initially
|
|
app-scoped (`default-provider-invalidation-flow`), which silently left Authentik's own session valid —
|
|
fixed by switching to `default-invalidation-flow`; and the provider had no `property_mappings`, so the
|
|
ID token never carried `preferred_username`/`email` and the UI showed the raw `sub` hash (with
|
|
`sub_mode: hashed_user_id`) next to the sign-out button, looking like a leaked session token — fixed by
|
|
attaching Authentik's default openid/profile/email scope mappings, no app code change needed. Full
|
|
writeup in `docs/oidc-setup.md`. The Authentik blueprint is now version-controlled at
|
|
`deploy/authentik/homelab-monitor-oidc.yaml` (source of truth — previously only lived on CT121 via
|
|
ad-hoc `scp`/`pct push`); redeploy steps are in `docs/oidc-setup.md`.
|
|
- **`e2e/`** (Playwright, done): drives a real browser against the **live** deployment, not a local dev
|
|
server. `local-login.spec.ts` and `oidc-login.spec.ts` — the latter uses a dedicated Authentik test
|
|
account (`playwright-test`, blueprint-provisioned, path `users/service-accounts`, never a real personal
|
|
login) so the full OIDC flow can run unattended. Not wired into CI (hits production, needs secrets) —
|
|
run by hand after auth-related changes. See `e2e/README.md` for the bugs it already caught.
|
|
- **Device discovery** (done, separate from the collector system above — it's inventory, not per-host
|
|
metrics): `scripts/discover-devices.sh` runs via systemd timer **on the CT122 host**, not in Docker
|
|
(real ARP entries live in the host's network namespace, not Docker's bridge network). API reads the
|
|
resulting JSON each poll cycle. Sortable by every column in the UI (click a header, click again to
|
|
reverse). Full reasoning in `docs/device-discovery.md`.
|
|
- **Identifying unknown devices** (done): five layers — MAC OUI vendor lookup (`mac-oui-lookup`,
|
|
computed on read), mDNS hostname (added to the discovery sweep), manual labels (new `device_labels`
|
|
table keyed by MAC, inline-editable in the dashboard, takes priority), an admin-triggered
|
|
**on-demand deep check** per unknown device (mDNS + targeted SSDP/UPnP + curated port scan + HTTP
|
|
title grab, runs on the CT122 host via a *parameterized* SSH forced-command reading
|
|
`$SSH_ORIGINAL_COMMAND`, strictly validated), and optional **Fingerbank** enrichment folded into that
|
|
same button (MAC + SSDP SERVER header, gated behind `FINGERBANK_API_KEY`, runs directly from the API
|
|
container — no host-level access needed, unlike the other deep-check steps). Full writeup in
|
|
`docs/device-discovery.md`. Already resolved ~59/71 devices including "Ring LLC" and multiple "Amazon
|
|
Technologies Inc." entries. Fingerbank's honest ceiling without DHCP fingerprint data (which we don't
|
|
have — not the DHCP server) is manufacturer-level confidence, same as free OUI — documented plainly,
|
|
not oversold; UI shows the confidence band so a low-confidence guess isn't mistaken for a confirmed ID.
|
|
Two real bugs hit while building this: `mac-oui-lookup`'s named export crashed the whole process at
|
|
startup under Node ESM/CJS interop (works fine under `require()`, fails under `import { x } from`,
|
|
confirmed by running the built output with `node`, not just `tsc`); and SQLite's `ALTER TABLE ADD
|
|
COLUMN` has no `IF NOT EXISTS` clause (conflated with `CREATE TABLE`'s support for it) — check
|
|
`pragma table_info` first instead.
|
|
- **New-device alerting** (done): `seen_macs` table (permanent, MAC-keyed, insert-only) tracks the first
|
|
time each device was ever seen; bootstrap-safe so turning this on doesn't alert on the whole existing
|
|
population. Pushes a Home Assistant `persistent_notification` when a genuinely new MAC appears
|
|
(`HOME_ASSISTANT_TOKEN` + `homeAssistant.url` in `hosts.yaml` — **token not yet provided by the user**,
|
|
so notifications aren't live yet even though detection and the dashboard badge are). Also shows a blue
|
|
"new" badge in the dashboard for anything first seen in the last 24h, independent of HA config. Real
|
|
bug caught via actual deployment (not just the unit test): the notification path was correctly
|
|
bootstrap-safe, but the UI badge wasn't bootstrap-aware at all — 65/71 devices showed as new on first
|
|
deploy. Fixed with a `seen_macs.is_bootstrap` column + a migration that backfills it correctly for
|
|
already-deployed data. Verified with a full synthetic end-to-end test (injected a fake device into the
|
|
discovery pipeline, confirmed only it got flagged, cleaned up after).
|
|
|
|
## Infrastructure this project owns
|
|
|
|
- **Gitea repo**: `jhodgkin/homelab-monitor`
|
|
- **Proxmox API token**: `monitor@pve!dashboard`, role `PVEAuditor` (read-only), created via `pveum` on
|
|
`pve`. Secret lives only in CT122's `.env`.
|
|
- **Deployment**: CT122 (`homelab-monitor`, 192.168.1.103), unprivileged LXC on `pve`, Docker + Compose.
|
|
SSH alias `homelab-monitor` in `~/.ssh/config` (root, key-based).
|
|
⚠️ **CT122 ID conflict**: the main homelab repo's `jhodgkin/homelab#15` (Grafana/Loki/Prometheus
|
|
migration) also planned to use CT122. That issue predates this dashboard's CT122 but wasn't checked
|
|
before provisioning. Flagged in a comment on that issue — when #15 is picked up, it needs a different
|
|
CT ID (123 or next free).
|
|
- **SSH collector key**: `monitor_ed25519` keypair lives only at `/opt/homelab-monitor/ssh/` on CT122
|
|
(gitignored, not in the repo). Public key installed on `omv` and `ripper` with a forced command —
|
|
see `docs/ssh-collector-key-setup.md`.
|
|
- **Authentik OIDC provider**: set up via an Authentik blueprint on CT121 (not a manual UI step) — see
|
|
`docs/oidc-setup.md`. Purely additive, didn't touch any existing Authentik user/credential. Issuer URL
|
|
is the public `https://auth.jerodrigged.com` (a pre-existing NPM proxy host, predates this project) —
|
|
**never point this at the LAN IP**, see the bug writeup above.
|
|
- **Authentik test account**: `playwright-test` (blueprint-provisioned, `authentik_core.user`, path
|
|
`users/service-accounts`), used only by `e2e/tests/oidc-login.spec.ts`. Not a real person's login.
|
|
- **Public exposure (done, issue #13 closed)**: `https://monitor.jerodrigged.com` is live. Chain is
|
|
Cloudflare Tunnel (existing tunnel `ac3a4440-...`, added a Public Hostname pointing at NPM, not
|
|
directly at the dashboard) → NPM proxy host id 14 (`192.168.1.185:80` → `192.168.1.103:8090`) → app.
|
|
NPM cert is Let's Encrypt (id 18). **`ssl_forced` must stay `false`** on this proxy host — Cloudflare
|
|
is in Flexible SSL mode (terminates TLS at the edge, talks plain HTTP to the tunnel origin), so
|
|
`ssl_forced: true` causes an infinite redirect loop. Hit this exact bug once already; don't re-enable it.
|
|
`COOKIE_SECURE` also stays `false` for the same underlying reason — the Fastify process never sees an
|
|
HTTPS connection even when the *browser* is on HTTPS, since TLS terminates before it.
|
|
- **Authentik redirect URIs**: both the public (`https://monitor.jerodrigged.com/...`, now primary in
|
|
`.env`) and LAN (`http://192.168.1.103:8090/...`) callback URLs are registered in the blueprint, so
|
|
OIDC login still works if the tunnel/DNS is ever down.
|
|
- **Vaultwarden**: three items saved — "Homelab Monitor (dashboard admin)" (local admin login),
|
|
"Homelab Monitor - Authentik OIDC client (homelab-monitor provider)" (client_id/secret, secure note),
|
|
and "Homelab Monitor - Authentik test account (playwright-test)" (the e2e test account's login).
|
|
- **Deep-check SSH key entry**: `monitor_ed25519`'s public key also has a **second, separate
|
|
`authorized_keys` entry on CT122's own root user** (not just omv/ripper) — forced command
|
|
`deep-check-device.sh`, parameterized via `$SSH_ORIGINAL_COMMAND` instead of the fixed-script pattern
|
|
used elsewhere. `deepCheck:` in `config/hosts.yaml` declares the target (CT122's own LAN IP).
|
|
- **Host packages** (one-time, not in any deploy script — a from-scratch CT122 re-provision would need
|
|
to redo this): `avahi-utils` and `miniupnpc`, installed via `apt-get` for mDNS/SSDP support.
|
|
|
|
## Zabbix: intentionally skipped, not blocked
|
|
|
|
Investigated CT109 while working issue #11 and found Zabbix was never actually finished being set up
|
|
(frontend not wired into Apache, setup wizard never run, no real hosts monitored, likely still on
|
|
default `Admin`/`zabbix` credentials). Turned out this is expected: `jhodgkin/homelab#15` already plans
|
|
to decommission Zabbix entirely in favor of Prometheus+Loki+Grafana. User's call: skip Zabbix
|
|
integration entirely rather than finish configuring a service that's being replaced. homelab-monitor
|
|
issue #11 is closed as won't-do — if a metrics-alerting panel is wanted later, it should target
|
|
Grafana/Prometheus once `homelab#15` lands, as a new issue, not a reopening of #11.
|
|
|
|
## Status: original backlog complete, OIDC verified end-to-end
|
|
|
|
All 14 issues filed at project start are closed (10 shipped, #11/Zabbix closed as won't-do, no others
|
|
skipped). `GET .../issues?state=open` on this repo should return empty. First real user click-through of
|
|
"Sign in with Authentik" surfaced two bugs (LAN-IP issuer, wrong-scheme redirect_uri in token exchange —
|
|
both above), fixed, and now covered by `e2e/oidc-login.spec.ts` as regression coverage. Both local and
|
|
OIDC login are Playwright-verified passing (5 consecutive runs) as of this writing.
|
|
|
|
e2e coverage expanded beyond the happy path (invalid credentials, session persistence, unauthenticated
|
|
API rejection, device labeling, deep-check, Fingerbank, new-device badge) — 16 tests across 9 spec
|
|
files, stable across repeated full-suite runs with parallel workers. All credentials this project
|
|
generated are in Vaultwarden except the Home Assistant token (see below, not provided yet).
|
|
|
|
Unknown-device identification (issue #15, then #16 Fingerbank, then #17 new-device alerting) shipped
|
|
after the OIDC work: OUI vendor lookup, mDNS, manual labels, on-demand deep check + Fingerbank, and
|
|
alerting on genuinely-new MACs. See the "Identifying unknown devices" and "New-device alerting"
|
|
architecture notes above.
|
|
|
|
If picking this up again with no specific ask from the user, there's no queued work — check in with them
|
|
for what's next rather than inventing scope. Remaining known gaps, none blocking:
|
|
|
|
- **`HOME_ASSISTANT_TOKEN` not yet set** — new-device detection and the dashboard badge both work now,
|
|
but the actual push notification to Home Assistant is silently skipped until this is provided (a
|
|
long-lived access token from HA's Profile > Security).
|
|
- `diskPaths` in `hosts.yaml` vs. the remote forced-command scripts on omv/ripper is a manual-sync point
|
|
(documented, not automated — only 2 hosts, hasn't been worth it).
|
|
- `e2e/` isn't wired into CI — run by hand (`cd e2e && npm test`) after any auth-related change.
|
|
- The two `avahi-utils`/`miniupnpc` host packages on CT122 aren't in any deploy script — see
|
|
"Infrastructure this project owns" above.
|
|
- User is evaluating buying a self-administrable router (UniFi/OPNsense) instead of the locked-down
|
|
GFiber one — if that happens, `knownDevices`/`deepCheck` IPs in `hosts.yaml` may need updating, and a
|
|
self-administered router might expose a real local API/DHCP data worth integrating (see conversation
|
|
history around 2026-07-13 for the GFiber investigation — no usable local API found for it specifically).
|