toknmtr — self-hostable Claude Code usage & analytics dashboard

Server (SvelteKit + SQLite Docker container) + per-machine agent that parses Claude Code JSONL transcripts. docker-compose one-command deploy; raw transcript view + search gated behind SHOW_TRANSCRIPTS (hidden by default). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08 19:33:55 +01:00 · 2026-07-08 19:33:55 +01:00 · 71a60ab054
commit 71a60ab054
74 changed files with 15613 additions and 0 deletions
--- a/ops/README.md
+++ b/ops/README.md
@ -0,0 +1,137 @@
+# ops/ — capture wiring (hooks + cron)
+
+This directory holds everything needed to make the `agent/` JSONL parser run automatically
+on a machine, instead of being invoked by hand. Two complementary triggers feed the same
+idempotent ingest pipeline:
+
+| Trigger                 | Script                                                                | Fires                       | Purpose                      |
+| ----------------------- | --------------------------------------------------------------------- | --------------------------- | ---------------------------- |
+| Claude Code `Stop` hook | `agent/hooks/toknmtr-capture.sh` (installed by `ops/install-hook.sh`) | end of every assistant turn | near-live capture            |
+| cron                    | `ops/install-cron.sh`                                                 | every ~10 min               | reconcile sweep / safety net |
+
+Both ultimately just run:
+
+```sh
+node --experimental-strip-types agent/run.ts --once
+```
+
+which is itself idempotent (see "Why both triggers are safe to overlap" below), so having
+both wired up is never a correctness problem — only a (very cheap) redundancy.
+
+## 1. One-time setup: `~/.toknmtr/env`
+
+Both the hook and the cron sweep read agent config from a file **outside**
+`~/.claude/settings.json`, specifically so `TOKNMTR_TOKEN` never ends up in a file that's
+more likely to be synced, shared, or pasted into a support thread.
+
+Create `~/.toknmtr/env` (plain `KEY=value` lines, shell-sourceable):
+
+```sh
+mkdir -p ~/.toknmtr
+cat > ~/.toknmtr/env <<'EOF'
+TOKNMTR_URL=http://<server-host>:3001
+TOKNMTR_TOKEN=<the API_TOKEN configured on the server>
+EOF
+chmod 600 ~/.toknmtr/env
+```
+
+- `TOKNMTR_URL` — base URL of the deployed server (see `.env.example` at the repo root for
+  the server-side `API_TOKEN` / `PORT` config that this must match).
+- `TOKNMTR_TOKEN` — must equal the server's `API_TOKEN` env var; sent as
+  `Authorization: Bearer <token>` on every `/api/ingest` request (`agent/push.ts`).
+
+Neither the hook script nor the installers will create this file for you — they treat its
+absence as "not configured yet" and (for the hook) silently no-op, or (for the installers)
+proceed anyway since the file only needs to exist by the time a sweep actually runs.
+
+If you'd rather keep config somewhere else, both scripts honor `TOKNMTR_ENV_FILE` to
+override the path (see each script's header for the full list of override env vars).
+
+## 2. Install the live (`Stop`-hook) capture path
+
+```sh
+ops/install-hook.sh
+```
+
+This merges a `Stop` hook entry into `~/.claude/settings.json` (via `jq`, additive — it
+will not touch any other hooks or settings already there) pointing at
+`agent/hooks/toknmtr-capture.sh`. It:
+
+- backs up `settings.json` (timestamped, alongside the original) before writing,
+- is idempotent — matches on the exact command string, so re-running is a no-op,
+- is reversible — `ops/install-hook.sh --remove` deletes just this entry.
+
+**Why `Stop` and not `SessionEnd`** (this is a real tradeoff, documented in full in the
+script's header comment — short version): `Stop` fires at the end of every assistant turn,
+so capture is close to live; `SessionEnd` only fires once when the CLI process exits, and
+Claude Code does not guarantee it fires on every exit path (e.g. a killed terminal can skip
+it). Since `agent/hooks/toknmtr-capture.sh` is designed to cost the foreground session
+basically nothing (it backgrounds the real work and returns in milliseconds — verified:
+~3ms wall time in testing), `Stop`'s higher firing frequency is nearly free, so it wins.
+
+The exact JSON this installs (for review before running it live) is reproduced in this
+repo's PR/report — see also `~/.claude/settings.json` directly after running the script.
+
+## 3. Install the cron reconcile sweep
+
+```sh
+ops/install-cron.sh
+```
+
+Adds a crontab line (default schedule `*/10 * * * *`) that sources `~/.toknmtr/env` and
+runs `agent/run.ts --once`, logging to `~/.toknmtr/cron.log`. Idempotent (matched by a
+`# toknmtr-cron-reconcile` marker comment) and reversible (`ops/install-cron.sh --remove`,
+or just delete the marked line via `crontab -e`). Only touches the one marked line — every
+other line in the user's crontab is preserved verbatim.
+
+## 4. Backfill (one-time, per machine)
+
+The hook/cron only push _new_ bytes appended to transcripts since the last run (see the
+per-file cursor logic in `agent/cursor.ts` / `agent/run.ts`). To ingest every transcript
+that already exists on a machine before capture was wired up:
+
+```sh
+cd ~/claude/projects/toknmtr
+node --experimental-strip-types agent/run.ts --backfill
+```
+
+`--backfill` ignores stored cursors and reparses every transcript from byte 0 (but still
+updates cursors afterward, so the _next_ run — hook or cron — resumes incrementally from
+there rather than re-walking everything again). Safe to re-run any time: ingest is an
+idempotent upsert keyed on `host + session_id + uuid` (events) and deduped on
+`session_id + message_id + request_id` (usage), so a repeated backfill just re-writes the
+same rows.
+
+## Why both triggers are safe to overlap
+
+Every push is an idempotent upsert. The event primary key is `host + session_id + uuid`
+(every physical JSONL line has a unique top-level `uuid`); usage rows are deduped to
+exactly one canonical row per `session_id + message_id + request_id` group server-side.
+So if a hook-triggered sweep and a cron-triggered sweep ever race (or a hook fires twice
+because of a fast back-to-back turn, or the cron sweep re-reads a tail the hook already
+pushed), the result is just redundant writes of already-correct rows — never duplicate or
+conflicting data. Cursors (`agent/cursor.ts`) are only advanced _after_ a push succeeds, so
+a failed/unreachable-server push is naturally retried by the next sweep (hook or cron)
+rather than silently dropping data.
+
+## Files in this directory
+
+- `install-hook.sh` — registers/unregisters the `Stop` hook in `~/.claude/settings.json`.
+- `install-cron.sh` — registers/unregisters the cron reconcile line.
+- `README.md` — this file.
+
+The hook script itself lives at `agent/hooks/toknmtr-capture.sh` (next to the rest of the
+agent code, since it's part of the agent's runtime surface, not an ops/deploy concern).
+
+## Troubleshooting
+
+- **Nothing showing up in the dashboard:** check `~/.toknmtr/capture.log` (hook) and
+  `~/.toknmtr/cron.log` (cron) for errors — most commonly a wrong `TOKNMTR_URL`/
+  `TOKNMTR_TOKEN` in `~/.toknmtr/env`, or the server being unreachable.
+- **Hook seems to never run:** confirm it's actually registered —
+  `jq '.hooks.Stop' ~/.claude/settings.json` — and that `agent/hooks/toknmtr-capture.sh` is
+  executable (`chmod +x`).
+- **Suspect a missed turn / gap:** the cron sweep will pick it up within its schedule
+  window regardless of what the hook did or didn't capture; you can also just run
+  `node --experimental-strip-types agent/run.ts --once` by hand at any time.
+- **Want to undo everything:** `ops/install-hook.sh --remove && ops/install-cron.sh --remove`.
--- a/ops/install-cron.sh
+++ b/ops/install-cron.sh
@ -0,0 +1,95 @@
+#!/usr/bin/env bash
+set -euo pipefail
+#
+# ops/install-cron.sh — idempotently install a crontab entry that runs the toknmtr agent in
+# reconcile mode (`--once`) every ~10 minutes.
+#
+# Why this exists alongside the Stop hook (ops/install-hook.sh):
+#   The hook is near-live but best-effort — it's deliberately fail-open (see
+#   agent/hooks/toknmtr-capture.sh), so it silently no-ops if the project dir/config/node
+#   binary is momentarily missing, and it never retries a failed push itself (a failed push
+#   just leaves the file's cursor unmoved, see agent/run.ts). It also only fires on Claude
+#   Code turns, so a transcript that's touched outside a tracked Stop event (or a session
+#   whose final hook fire was the one that raced/failed) won't get its tail captured until
+#   *something* sweeps again. This cron line is that backstop: a periodic, host-crontab-
+#   driven sweep that's independent of any particular Claude Code session being open, so the
+#   data self-heals even if every hook fire that session ever missed/failed.
+#   Push is idempotent (PK = host+session_id+uuid, see CLAUDE.md), so overlapping
+#   hook-triggered and cron-triggered sweeps are always safe to interleave.
+#
+# What this script does:
+#   - Adds (or updates, if already present) a single crontab line running
+#     `node --experimental-strip-types agent/run.ts --once` every 10 minutes, sourcing
+#     ~/.toknmtr/env first for TOKNMTR_URL/TOKNMTR_TOKEN, with stdout/stderr appended to a
+#     log file.
+#   - Idempotent: matches by a unique marker comment, so re-running replaces (rather than
+#     duplicating) the line — safe to re-run after changing TOKNMTR_PROJECT_DIR etc.
+#   - Reversible: `ops/install-cron.sh --remove` deletes just the marked line; everything
+#     else in the user's crontab is left untouched.
+#
+# Usage:
+#   ops/install-cron.sh            # install/update the cron line
+#   ops/install-cron.sh --remove   # remove just this cron line
+#
+# Override knobs (env, all optional):
+#   TOKNMTR_PROJECT_DIR   path to the toknmtr repo checkout (default: ~/claude/projects/toknmtr)
+#   TOKNMTR_ENV_FILE      path to the config file            (default: ~/.toknmtr/env)
+#   TOKNMTR_LOG_FILE      cron sweep log path                (default: ~/.toknmtr/cron.log)
+#   TOKNMTR_CRON_SCHEDULE cron schedule expression            (default: "*/10 * * * *")
+
+PROJECT_DIR="${TOKNMTR_PROJECT_DIR:-$HOME/claude/projects/toknmtr}"
+CONFIG_FILE="${TOKNMTR_ENV_FILE:-$HOME/.toknmtr/env}"
+LOG_FILE="${TOKNMTR_LOG_FILE:-$HOME/.toknmtr/cron.log}"
+SCHEDULE="${TOKNMTR_CRON_SCHEDULE:-*/10 * * * *}"
+
+MARKER="# toknmtr-cron-reconcile"
+CRON_CMD="cd \"$PROJECT_DIR\" && . \"$CONFIG_FILE\" && node --experimental-strip-types agent/run.ts --once >> \"$LOG_FILE\" 2>&1"
+CRON_LINE="$SCHEDULE bash -lc '$CRON_CMD' $MARKER"
+
+current_crontab() {
+	crontab -l 2>/dev/null || true
+}
+
+if [ "${1:-}" = "--remove" ]; then
+	existing="$(current_crontab)"
+	if ! printf '%s\n' "$existing" | grep -qF "$MARKER"; then
+		echo "no toknmtr cron line found, nothing to remove"
+		exit 0
+	fi
+	printf '%s\n' "$existing" | grep -vF "$MARKER" | crontab -
+	echo "removed toknmtr reconcile cron line"
+	exit 0
+fi
+
+command -v crontab >/dev/null 2>&1 || {
+	echo "error: crontab is required (e.g. 'sudo apt install cron') and the cron daemon must be running" >&2
+	exit 1
+}
+[ -d "$PROJECT_DIR" ] || {
+	echo "error: $PROJECT_DIR does not exist (set TOKNMTR_PROJECT_DIR?)" >&2
+	exit 1
+}
+[ -f "$PROJECT_DIR/agent/run.ts" ] || {
+	echo "error: $PROJECT_DIR/agent/run.ts not found — is TOKNMTR_PROJECT_DIR correct?" >&2
+	exit 1
+}
+
+mkdir -p "$(dirname "$LOG_FILE")"
+
+# Replace any prior toknmtr line (matched by marker) with the new one; append if absent.
+existing="$(current_crontab)"
+filtered="$(printf '%s\n' "$existing" | grep -vF "$MARKER" || true)"
+{
+	[ -n "$filtered" ] && printf '%s\n' "$filtered"
+	printf '%s\n' "$CRON_LINE"
+} | crontab -
+
+echo "installed toknmtr reconcile cron line (schedule: $SCHEDULE):"
+echo "  $CRON_LINE"
+echo ""
+echo "config read from: $CONFIG_FILE (must exist with TOKNMTR_URL / TOKNMTR_TOKEN, see ops/README.md)"
+echo "sweep log:        $LOG_FILE"
+echo ""
+echo "to remove later:"
+echo "  ops/install-cron.sh --remove"
+echo "  (or run 'crontab -e' and delete the line ending in '$MARKER')"
--- a/ops/install-hook.sh
+++ b/ops/install-hook.sh
@ -0,0 +1,113 @@
+#!/usr/bin/env bash
+set -euo pipefail
+#
+# ops/install-hook.sh — idempotently register the toknmtr capture hook
+# (agent/hooks/toknmtr-capture.sh) in ~/.claude/settings.json under the Claude Code 'Stop'
+# hook event.
+#
+# WHY 'Stop' AND NOT 'SessionEnd':
+#   Stop fires at the end of EVERY assistant turn (each time Claude finishes responding and
+#   yields back to the user) — not just once when the whole CLI process exits. That makes
+#   capture close to live: a dashboard refresh a few seconds after a turn already reflects
+#   it. The cost is the hook running more often, but agent/hooks/toknmtr-capture.sh is
+#   designed to be near-zero-cost on the foreground session (it backgrounds the real work
+#   and returns in milliseconds — see that script's header), so the extra frequency is
+#   nearly free.
+#   SessionEnd only fires once, when the CLI process actually exits — on a long interactive
+#   session that can be hours after the data was generated, and Claude Code does not
+#   guarantee SessionEnd fires on every exit path (e.g. a killed/crashed terminal can skip
+#   it entirely), so relying on it alone risks losing the tail of a session permanently
+#   until the next cron sweep (ops/install-cron.sh) catches it.
+#   Net: Stop is the better default given the hook is cheap. The cron sweep is an
+#   independent safety net either way (server-down resilience, missed hook fires, etc.) —
+#   it doesn't change this tradeoff, it just makes the choice lower-stakes. Revisit by
+#   swapping HOOK_EVENT below to "SessionEnd" if Stop's per-turn frequency ever proves to be
+#   a real problem in practice (e.g. very chatty multi-turn sessions on a slow machine).
+#
+# What this script does:
+#   - Merges (via jq) a Stop-hook entry pointing at agent/hooks/toknmtr-capture.sh into
+#     ~/.claude/settings.json, WITHOUT touching any other keys/hooks already in that file.
+#   - Idempotent: matches on the exact command string, so re-running is a safe no-op.
+#   - Writes a timestamped backup of settings.json before every modification.
+#   - Reversible: `ops/install-hook.sh --remove` undoes just this hook's entry (or restore
+#     a backup manually — see the "to remove later" line this script prints on install).
+#
+# Usage:
+#   ops/install-hook.sh            # install/update the hook
+#   ops/install-hook.sh --remove   # remove this hook's entry from settings.json
+#
+# Override knobs (env, all optional):
+#   TOKNMTR_PROJECT_DIR    path to the toknmtr repo checkout (default: ~/claude/projects/toknmtr)
+#   CLAUDE_SETTINGS_FILE   path to settings.json             (default: ~/.claude/settings.json)
+
+PROJECT_DIR="${TOKNMTR_PROJECT_DIR:-$HOME/claude/projects/toknmtr}"
+SETTINGS_FILE="${CLAUDE_SETTINGS_FILE:-$HOME/.claude/settings.json}"
+HOOK_SCRIPT="$PROJECT_DIR/agent/hooks/toknmtr-capture.sh"
+HOOK_EVENT="Stop"
+HOOK_CMD="bash \"$HOOK_SCRIPT\""
+HOOK_TIMEOUT_S=10
+
+command -v jq >/dev/null 2>&1 || {
+	echo "error: jq is required (sudo apt install jq / brew install jq)" >&2
+	exit 1
+}
+
+backup_settings() {
+	[ -f "$SETTINGS_FILE" ] || return 0
+	local stamp
+	stamp="$(date +%Y%m%dT%H%M%S)"
+	cp "$SETTINGS_FILE" "$SETTINGS_FILE.bak.$stamp"
+	echo "backed up $SETTINGS_FILE -> $SETTINGS_FILE.bak.$stamp"
+}
+
+if [ "${1:-}" = "--remove" ]; then
+	if [ ! -f "$SETTINGS_FILE" ]; then
+		echo "no settings file at $SETTINGS_FILE, nothing to remove"
+		exit 0
+	fi
+	backup_settings
+	tmp="$(mktemp)"
+	jq --arg cmd "$HOOK_CMD" --arg event "$HOOK_EVENT" '
+    if (.hooks // {} | has($event)) then
+      .hooks[$event] = [
+        .hooks[$event][] |
+        .hooks = [(.hooks // [])[] | select(.command != $cmd)] |
+        select((.hooks | length) > 0)
+      ]
+    else . end
+  ' "$SETTINGS_FILE" >"$tmp" && mv "$tmp" "$SETTINGS_FILE"
+	echo "removed toknmtr capture hook from $SETTINGS_FILE.$HOOK_EVENT (backup written above; other $HOOK_EVENT hooks, if any, were left untouched)"
+	exit 0
+fi
+
+if [ ! -f "$HOOK_SCRIPT" ]; then
+	echo "error: $HOOK_SCRIPT not found (is TOKNMTR_PROJECT_DIR=$PROJECT_DIR correct?)" >&2
+	exit 1
+fi
+chmod +x "$HOOK_SCRIPT" 2>/dev/null || true
+
+mkdir -p "$(dirname "$SETTINGS_FILE")"
+[ -f "$SETTINGS_FILE" ] || echo '{}' >"$SETTINGS_FILE"
+jq empty "$SETTINGS_FILE" || {
+	echo "error: $SETTINGS_FILE is not valid JSON — fix it manually before running this script" >&2
+	exit 1
+}
+
+backup_settings
+
+tmp="$(mktemp)"
+jq --arg cmd "$HOOK_CMD" --arg event "$HOOK_EVENT" --argjson timeout "$HOOK_TIMEOUT_S" '
+  .hooks //= {} |
+  .hooks[$event] //= [] |
+  ([.hooks[$event][]? | (.hooks // [])[]? | .command] | index($cmd)) as $already |
+  if $already == null then
+    .hooks[$event] += [{"hooks": [{"type": "command", "command": $cmd, "timeout": $timeout}]}]
+  else . end
+' "$SETTINGS_FILE" >"$tmp" && mv "$tmp" "$SETTINGS_FILE"
+
+echo "installed toknmtr capture hook into $SETTINGS_FILE under hooks.$HOOK_EVENT"
+echo "  command: $HOOK_CMD"
+echo ""
+echo "to remove later:"
+echo "  ops/install-hook.sh --remove"
+echo "  (or restore a $SETTINGS_FILE.bak.<timestamp> file written next to it)"