Research

Digital Surface Labs

"Fleet Live Audit"

2026-04-10

Fleet Live Audit — 2026-04-10

What Changed

This pass added a shared fleet control plane under sync/fleet/:

  • inventory.json for current hosts and core services
  • fleetctl.py for status, verify, logs, restart, and deploy
  • standardized polling deploy worker scripts/git_autodeploy.sh
  • direct-tunnel example configs for prometheus, atlas, spark, epimetheus
  • Atlas rollout package for clean autodeploy timers and env files
  • Live Atlas migration so buddy.digitalsurfacelabs.com and health.digitalsurfacelabs.com now route through Atlas's own tunnel targets

Live Findings

Prometheus

  • Reachable over SSH
  • cloudflared active
  • docker active
  • nginx active after removing stale local ownership of buddy and health
  • memex-server active
  • memex-chromadb active
  • no longer owns live buddy or health
  • now proxies Spark public hostnames (audio, spark, forge, llama) to the current Spark LAN address

Atlas

  • Reachable over SSH
  • cloudflared active
  • nginx active
  • buddy.service enabled and active
  • health.service enabled and active
  • fleet-git-autodeploy@buddy.timer enabled and healthy
  • fleet-git-autodeploy@health.timer enabled and healthy
  • Atlas tunnel now points buddy to localhost:8001 and health to localhost:8104
  • Atlas allows passwordless sudo, so it is the first host migrated onto the cleaned-up deploy path

Epimetheus

  • Reachable over SSH
  • Lightly loaded
  • Current owner for ramp, prospector, and soothsayer
  • ramp-dashboard, ramp-stats, prospector, and soothsayer all active
  • prospector is on a git checkout and now has an active fleet-git-autodeploy@prospector.timer

Spark

  • Reachable over SSH
  • docker active
  • local endpoints on 8090, 8200, and 11434 respond successfully
  • legacy /opt/spark-api systemd units are not the active runtime path
  • public audio, spark, forge, and llama hostnames now return 200
  • public Spark access still depends on the Prometheus tunnel rather than a direct Spark-owned tunnel

Current Verify Output

python3 fleet/fleetctl.py verify is expected to pass once the shared inventory is updated to reflect the live ownership model:

  • ramp and prospector belong to epimetheus, not prometheus
  • Spark should be represented by the live DGX endpoints, not the stale spark-api.service name

Remaining Manual Steps

  • Prometheus: either grant a safe passwordless sudo path for specific restart commands or change the systemd deploy services to run with the privileges they actually need.
  • Spark: install a direct Spark tunnel and the cleaned-up autodeploy units once privileged access is available.
  • Memex: convert /ssd/memex into a tracked git deployment path if automatic deploys are desired there.