146 lines
12 KiB
Markdown
146 lines
12 KiB
Markdown
# Homelab Monitor — Project Memory
|
|
|
|
Unified health dashboard for the homelab (infra docs: `jhodgkin/homelab` repo, `docs/infrastructure.md`).
|
|
One login instead of logging into Proxmox, Zabbix, OMV, and every service separately.
|
|
|
|
## How to resume work in a new session
|
|
|
|
1. List open issues: `GET /api/v1/repos/jhodgkin/homelab-monitor/issues?state=open&limit=30` on
|
|
`git.jerodrigged.com` (same pattern as the main homelab backlog, see global `~/.claude/CLAUDE.md`).
|
|
Milestone `v1-dashboard` is the current focus; everything else is backlog.
|
|
2. Pick an issue, comment that you're starting it, do the work, comment what you did, close it.
|
|
3. Commit and push after every meaningful step — don't let work sit uncommitted.
|
|
|
|
## Architecture
|
|
|
|
- `apps/api` — Fastify + TypeScript. Polls **collectors** on an interval (`POLL_INTERVAL_SECONDS`,
|
|
default 30s), writes snapshots to SQLite (`apps/api/src/db`), serves `/api/*`.
|
|
- `apps/web` — React + Vite + TypeScript. Polls `/api/hosts` every 15s, renders host/container cards.
|
|
- `config/hosts.yaml` — declares *what* to monitor (no secrets, committed). Credentials live in `.env`
|
|
(gitignored) — see `.env.example` for the full list.
|
|
- Collectors implement the `Collector` interface (`apps/api/src/collectors/types.ts`) and are registered
|
|
in `apps/api/src/index.ts`. Adding a new data source = new collector + config entry, no other wiring.
|
|
- `ProxmoxCollector` (done): one API call to `pve` returns CPU/mem/disk **and PSI pressure**
|
|
(`pressurecpusome`, `pressurememoryfull`) for the host + every LXC. Covers ~22 of ~24 machines.
|
|
- `SshHostCollector` (done): covers `.180` (omv) and `.171` (ripper), the bare-metal boxes Proxmox
|
|
can't see. Dedicated SSH key with a **forced `command=`** in the remote `authorized_keys` — the
|
|
server always runs a fixed read-only script regardless of what's exec'd, so a leaked key is still
|
|
contained. See `docs/ssh-collector-key-setup.md`, including the gotcha that `diskPaths` in
|
|
`hosts.yaml` must be kept in sync **by hand** with the remote script's `DISK_<label>=` lines.
|
|
- No Zabbix collector — see "Zabbix: intentionally skipped" below.
|
|
- **Historical sparklines** (done): `/api/hosts` embeds the last ~40 samples per host (one windowed
|
|
query, `ROW_NUMBER() OVER (PARTITION BY host_id ...)`), rendered as small hand-rolled SVG sparklines
|
|
in each card — no charting library. Fixed a real bug while building this: the Proxmox **host node's**
|
|
own mem/disk had been `null` the whole time (`/nodes/{node}/status` nests `memory`/`rootfs` objects;
|
|
code assumed the flat shape the *LXC listing* endpoint uses). LXC entries were never affected.
|
|
- **Hot-reload** (done): `sshHosts`/`knownDevices` in `config/hosts.yaml` are re-read on the next poll
|
|
cycle (mtime check) — no container restart needed. Proxmox hosts already needed no config (auto-
|
|
discovered every poll). See `docs/hot-reload.md` — includes a real single-file-bind-mount inode bug
|
|
that silently broke this on first deploy, fixed by mounting `./config` as a directory.
|
|
- Auth: local (bcrypt + signed session cookie) is **always** available — it's not an exclusive mode
|
|
switch. OIDC via Authentik is an *additional* sign-in button, live when `OIDC_ENABLED=true`. See
|
|
`docs/oidc-setup.md`. Deliberately additive so OIDC config problems can never lock out the admin login.
|
|
Two real bugs found only once someone actually clicked the button (fixed, see `docs/oidc-setup.md` and
|
|
`e2e/README.md`): `OIDC_ISSUER_URL` pointed at Authentik's LAN IP, which got baked into the
|
|
browser-facing `authorization_endpoint`; and the callback hardcoded `http://` when reconstructing the
|
|
current URL (Fastify never sees HTTPS, it terminates upstream), sending the wrong scheme as
|
|
`redirect_uri` in the token exchange. Both are exactly the class of bug curl-based verification cannot
|
|
catch — real redirect chains and a real IdP login form need a real browser.
|
|
- **`e2e/`** (Playwright, done): drives a real browser against the **live** deployment, not a local dev
|
|
server. `local-login.spec.ts` and `oidc-login.spec.ts` — the latter uses a dedicated Authentik test
|
|
account (`playwright-test`, blueprint-provisioned, path `users/service-accounts`, never a real personal
|
|
login) so the full OIDC flow can run unattended. Not wired into CI (hits production, needs secrets) —
|
|
run by hand after auth-related changes. See `e2e/README.md` for the bugs it already caught.
|
|
- **Device discovery** (done, separate from the collector system above — it's inventory, not per-host
|
|
metrics): `scripts/discover-devices.sh` runs via systemd timer **on the CT122 host**, not in Docker
|
|
(real ARP entries live in the host's network namespace, not Docker's bridge network). API reads the
|
|
resulting JSON each poll cycle. Sortable by every column in the UI (click a header, click again to
|
|
reverse). Full reasoning in `docs/device-discovery.md`.
|
|
- **Identifying unknown devices** (done): four layers — MAC OUI vendor lookup (`mac-oui-lookup`,
|
|
computed on read), mDNS hostname (added to the discovery sweep), manual labels (new `device_labels`
|
|
table keyed by MAC, inline-editable in the dashboard, takes priority), and an admin-triggered
|
|
**on-demand deep check** per unknown device (mDNS + targeted SSDP/UPnP + curated port scan + HTTP
|
|
title grab, runs on the CT122 host via a *parameterized* SSH forced-command reading
|
|
`$SSH_ORIGINAL_COMMAND`, strictly validated). Full writeup in `docs/device-discovery.md`. Already
|
|
resolved ~59/71 devices including "Ring LLC" and multiple "Amazon Technologies Inc." entries.
|
|
Two real bugs hit while building this: `mac-oui-lookup`'s named export crashed the whole process at
|
|
startup under Node ESM/CJS interop (works fine under `require()`, fails under `import { x } from`,
|
|
confirmed by running the built output with `node`, not just `tsc`); and SQLite's `ALTER TABLE ADD
|
|
COLUMN` has no `IF NOT EXISTS` clause (conflated with `CREATE TABLE`'s support for it) — check
|
|
`pragma table_info` first instead.
|
|
|
|
## Infrastructure this project owns
|
|
|
|
- **Gitea repo**: `jhodgkin/homelab-monitor`
|
|
- **Proxmox API token**: `monitor@pve!dashboard`, role `PVEAuditor` (read-only), created via `pveum` on
|
|
`pve`. Secret lives only in CT122's `.env`.
|
|
- **Deployment**: CT122 (`homelab-monitor`, 192.168.1.103), unprivileged LXC on `pve`, Docker + Compose.
|
|
SSH alias `homelab-monitor` in `~/.ssh/config` (root, key-based).
|
|
⚠️ **CT122 ID conflict**: the main homelab repo's `jhodgkin/homelab#15` (Grafana/Loki/Prometheus
|
|
migration) also planned to use CT122. That issue predates this dashboard's CT122 but wasn't checked
|
|
before provisioning. Flagged in a comment on that issue — when #15 is picked up, it needs a different
|
|
CT ID (123 or next free).
|
|
- **SSH collector key**: `monitor_ed25519` keypair lives only at `/opt/homelab-monitor/ssh/` on CT122
|
|
(gitignored, not in the repo). Public key installed on `omv` and `ripper` with a forced command —
|
|
see `docs/ssh-collector-key-setup.md`.
|
|
- **Authentik OIDC provider**: set up via an Authentik blueprint on CT121 (not a manual UI step) — see
|
|
`docs/oidc-setup.md`. Purely additive, didn't touch any existing Authentik user/credential. Issuer URL
|
|
is the public `https://auth.jerodrigged.com` (a pre-existing NPM proxy host, predates this project) —
|
|
**never point this at the LAN IP**, see the bug writeup above.
|
|
- **Authentik test account**: `playwright-test` (blueprint-provisioned, `authentik_core.user`, path
|
|
`users/service-accounts`), used only by `e2e/tests/oidc-login.spec.ts`. Not a real person's login.
|
|
- **Public exposure (done, issue #13 closed)**: `https://monitor.jerodrigged.com` is live. Chain is
|
|
Cloudflare Tunnel (existing tunnel `ac3a4440-...`, added a Public Hostname pointing at NPM, not
|
|
directly at the dashboard) → NPM proxy host id 14 (`192.168.1.185:80` → `192.168.1.103:8090`) → app.
|
|
NPM cert is Let's Encrypt (id 18). **`ssl_forced` must stay `false`** on this proxy host — Cloudflare
|
|
is in Flexible SSL mode (terminates TLS at the edge, talks plain HTTP to the tunnel origin), so
|
|
`ssl_forced: true` causes an infinite redirect loop. Hit this exact bug once already; don't re-enable it.
|
|
`COOKIE_SECURE` also stays `false` for the same underlying reason — the Fastify process never sees an
|
|
HTTPS connection even when the *browser* is on HTTPS, since TLS terminates before it.
|
|
- **Authentik redirect URIs**: both the public (`https://monitor.jerodrigged.com/...`, now primary in
|
|
`.env`) and LAN (`http://192.168.1.103:8090/...`) callback URLs are registered in the blueprint, so
|
|
OIDC login still works if the tunnel/DNS is ever down.
|
|
- **Vaultwarden**: three items saved — "Homelab Monitor (dashboard admin)" (local admin login),
|
|
"Homelab Monitor - Authentik OIDC client (homelab-monitor provider)" (client_id/secret, secure note),
|
|
and "Homelab Monitor - Authentik test account (playwright-test)" (the e2e test account's login).
|
|
- **Deep-check SSH key entry**: `monitor_ed25519`'s public key also has a **second, separate
|
|
`authorized_keys` entry on CT122's own root user** (not just omv/ripper) — forced command
|
|
`deep-check-device.sh`, parameterized via `$SSH_ORIGINAL_COMMAND` instead of the fixed-script pattern
|
|
used elsewhere. `deepCheck:` in `config/hosts.yaml` declares the target (CT122's own LAN IP).
|
|
- **Host packages** (one-time, not in any deploy script — a from-scratch CT122 re-provision would need
|
|
to redo this): `avahi-utils` and `miniupnpc`, installed via `apt-get` for mDNS/SSDP support.
|
|
|
|
## Zabbix: intentionally skipped, not blocked
|
|
|
|
Investigated CT109 while working issue #11 and found Zabbix was never actually finished being set up
|
|
(frontend not wired into Apache, setup wizard never run, no real hosts monitored, likely still on
|
|
default `Admin`/`zabbix` credentials). Turned out this is expected: `jhodgkin/homelab#15` already plans
|
|
to decommission Zabbix entirely in favor of Prometheus+Loki+Grafana. User's call: skip Zabbix
|
|
integration entirely rather than finish configuring a service that's being replaced. homelab-monitor
|
|
issue #11 is closed as won't-do — if a metrics-alerting panel is wanted later, it should target
|
|
Grafana/Prometheus once `homelab#15` lands, as a new issue, not a reopening of #11.
|
|
|
|
## Status: original backlog complete, OIDC verified end-to-end
|
|
|
|
All 14 issues filed at project start are closed (10 shipped, #11/Zabbix closed as won't-do, no others
|
|
skipped). `GET .../issues?state=open` on this repo should return empty. First real user click-through of
|
|
"Sign in with Authentik" surfaced two bugs (LAN-IP issuer, wrong-scheme redirect_uri in token exchange —
|
|
both above), fixed, and now covered by `e2e/oidc-login.spec.ts` as regression coverage. Both local and
|
|
OIDC login are Playwright-verified passing (5 consecutive runs) as of this writing.
|
|
|
|
e2e coverage expanded beyond the happy path (invalid credentials, session persistence, unauthenticated
|
|
API rejection, device labeling, deep-check) — 13 tests across 6 spec files, stable across repeated
|
|
full-suite runs with parallel workers. All credentials this project generated are in Vaultwarden.
|
|
|
|
Unknown-device identification (issue #15) shipped after the OIDC work: OUI vendor lookup, mDNS,
|
|
manual labels, on-demand deep check. See the "Identifying unknown devices" architecture note above.
|
|
|
|
If picking this up again with no specific ask from the user, there's no queued work — check in with them
|
|
for what's next rather than inventing scope. Remaining known gaps, none blocking:
|
|
|
|
- `diskPaths` in `hosts.yaml` vs. the remote forced-command scripts on omv/ripper is a manual-sync point
|
|
(documented, not automated — only 2 hosts, hasn't been worth it).
|
|
- `e2e/` isn't wired into CI — run by hand (`cd e2e && npm test`) after any auth-related change.
|
|
- The two `avahi-utils`/`miniupnpc` host packages on CT122 aren't in any deploy script — see
|
|
"Infrastructure this project owns" above.
|