toknmtr-selfhost/ops/README.md

# ops/ — capture wiring (hooks + cron)

This directory holds everything needed to make the `agent/` JSONL parser run automatically
on a machine, instead of being invoked by hand. Two complementary triggers feed the same
idempotent ingest pipeline:

| Trigger                 | Script                                                                | Fires                       | Purpose                      |
| ----------------------- | --------------------------------------------------------------------- | --------------------------- | ---------------------------- |
| Claude Code `Stop` hook | `agent/hooks/toknmtr-capture.sh` (installed by `ops/install-hook.sh`) | end of every assistant turn | near-live capture            |
| cron                    | `ops/install-cron.sh`                                                 | every ~10 min               | reconcile sweep / safety net |

Both ultimately just run:

```sh
node --experimental-strip-types agent/run.ts --once
```

which is itself idempotent (see "Why both triggers are safe to overlap" below), so having
both wired up is never a correctness problem — only a (very cheap) redundancy.

## 1. One-time setup: `~/.toknmtr/env`

Both the hook and the cron sweep read agent config from a file **outside**
`~/.claude/settings.json`, specifically so `TOKNMTR_TOKEN` never ends up in a file that's
more likely to be synced, shared, or pasted into a support thread.

Create `~/.toknmtr/env` (plain `KEY=value` lines, shell-sourceable):

```sh
mkdir -p ~/.toknmtr
cat > ~/.toknmtr/env <<'EOF'
TOKNMTR_URL=http://<server-host>:3001
TOKNMTR_TOKEN=<the API_TOKEN configured on the server>
EOF
chmod 600 ~/.toknmtr/env
```

- `TOKNMTR_URL` — base URL of the deployed server (see `.env.example` at the repo root for
  the server-side `API_TOKEN` / `PORT` config that this must match).
- `TOKNMTR_TOKEN` — must equal the server's `API_TOKEN` env var; sent as
  `Authorization: Bearer <token>` on every `/api/ingest` request (`agent/push.ts`).

Neither the hook script nor the installers will create this file for you — they treat its
absence as "not configured yet" and (for the hook) silently no-op, or (for the installers)
proceed anyway since the file only needs to exist by the time a sweep actually runs.

If you'd rather keep config somewhere else, both scripts honor `TOKNMTR_ENV_FILE` to
override the path (see each script's header for the full list of override env vars).

## 2. Install the live (`Stop`-hook) capture path

```sh
ops/install-hook.sh
```

This merges a `Stop` hook entry into `~/.claude/settings.json` (via `jq`, additive — it
will not touch any other hooks or settings already there) pointing at
`agent/hooks/toknmtr-capture.sh`. It:

- backs up `settings.json` (timestamped, alongside the original) before writing,
- is idempotent — matches on the exact command string, so re-running is a no-op,
- is reversible — `ops/install-hook.sh --remove` deletes just this entry.

**Why `Stop` and not `SessionEnd`** (this is a real tradeoff, documented in full in the
script's header comment — short version): `Stop` fires at the end of every assistant turn,
so capture is close to live; `SessionEnd` only fires once when the CLI process exits, and
Claude Code does not guarantee it fires on every exit path (e.g. a killed terminal can skip
it). Since `agent/hooks/toknmtr-capture.sh` is designed to cost the foreground session
basically nothing (it backgrounds the real work and returns in milliseconds — verified:
~3ms wall time in testing), `Stop`'s higher firing frequency is nearly free, so it wins.

The exact JSON this installs (for review before running it live) is reproduced in this
repo's PR/report — see also `~/.claude/settings.json` directly after running the script.

## 3. Install the cron reconcile sweep

```sh
ops/install-cron.sh
```

Adds a crontab line (default schedule `*/10 * * * *`) that sources `~/.toknmtr/env` and
runs `agent/run.ts --once`, logging to `~/.toknmtr/cron.log`. Idempotent (matched by a
`# toknmtr-cron-reconcile` marker comment) and reversible (`ops/install-cron.sh --remove`,
or just delete the marked line via `crontab -e`). Only touches the one marked line — every
other line in the user's crontab is preserved verbatim.

## 4. Backfill (one-time, per machine)

The hook/cron only push _new_ bytes appended to transcripts since the last run (see the
per-file cursor logic in `agent/cursor.ts` / `agent/run.ts`). To ingest every transcript
that already exists on a machine before capture was wired up:

```sh
cd ~/claude/projects/toknmtr
node --experimental-strip-types agent/run.ts --backfill
```

`--backfill` ignores stored cursors and reparses every transcript from byte 0 (but still
updates cursors afterward, so the _next_ run — hook or cron — resumes incrementally from
there rather than re-walking everything again). Safe to re-run any time: ingest is an
idempotent upsert keyed on `host + session_id + uuid` (events) and deduped on
`session_id + message_id + request_id` (usage), so a repeated backfill just re-writes the
same rows.

## Why both triggers are safe to overlap

Every push is an idempotent upsert. The event primary key is `host + session_id + uuid`
(every physical JSONL line has a unique top-level `uuid`); usage rows are deduped to
exactly one canonical row per `session_id + message_id + request_id` group server-side.
So if a hook-triggered sweep and a cron-triggered sweep ever race (or a hook fires twice
because of a fast back-to-back turn, or the cron sweep re-reads a tail the hook already
pushed), the result is just redundant writes of already-correct rows — never duplicate or
conflicting data. Cursors (`agent/cursor.ts`) are only advanced _after_ a push succeeds, so
a failed/unreachable-server push is naturally retried by the next sweep (hook or cron)
rather than silently dropping data.

## Files in this directory

- `install-hook.sh` — registers/unregisters the `Stop` hook in `~/.claude/settings.json`.
- `install-cron.sh` — registers/unregisters the cron reconcile line.
- `README.md` — this file.

The hook script itself lives at `agent/hooks/toknmtr-capture.sh` (next to the rest of the
agent code, since it's part of the agent's runtime surface, not an ops/deploy concern).

## Troubleshooting

- **Nothing showing up in the dashboard:** check `~/.toknmtr/capture.log` (hook) and
  `~/.toknmtr/cron.log` (cron) for errors — most commonly a wrong `TOKNMTR_URL`/
  `TOKNMTR_TOKEN` in `~/.toknmtr/env`, or the server being unreachable.
- **Hook seems to never run:** confirm it's actually registered —
  `jq '.hooks.Stop' ~/.claude/settings.json` — and that `agent/hooks/toknmtr-capture.sh` is
  executable (`chmod +x`).
- **Suspect a missed turn / gap:** the cron sweep will pick it up within its schedule
  window regardless of what the hook did or didn't capture; you can also just run
  `node --experimental-strip-types agent/run.ts --once` by hand at any time.
- **Want to undo everything:** `ops/install-hook.sh --remove && ops/install-cron.sh --remove`.