Stormin' The Castle

vibr8agentsfine-tuningsovereigntytrace-forge

Own Your Own Agent Traces

by John Robinson @johnrobinsn

Trace archive

This is your data.

Every prompt your coding agent sees, every file it reads, every reasoning step it works through on your problem — all of it represents your attention, your time, your judgment. You're the raw ingredient that actually make the agents solve the problems that you care about.

If you don't capture it, curate it, and use it, you're giving all of that away.

Here's a question you probably haven't asked yourself: where did it go?

You've been in a Claude Code session for an hour. You've asked it to grep for something, read five files, write a migration, run the tests, fix the one that broke. Twenty-plus tool calls. A rough shape of a problem that became a working change.

Every prompt — including the parts of your private code that got loaded into the context. Every tool call — including the ones you ran against your production database. Every response — including the parts where the model worked out something that was yours. All of it went to a server you don't own, and nothing I describe below changes that. The vendor's traffic doesn't stop. Their retention policy doesn't change. Whatever their terms say about training on your activity still applies.

What it does change is whether you also have the record — at wire-level fidelity, on your disk, under your retention policy, available when their terms change, when you want to feed it to a different model, or when you want to train on it yourself.

That's the lede. The rest of this post is about what I did about it for myself.

The why, before the how

If you're going to run coding agents daily — I run between six and ten of them concurrently through vibr8 — you should own three things:

This piece is a companion to the project I've been quietly building — vibr8 — the "really your own" agent host that ships v0.1 in a couple of weeks. Running agents on your own hardware is downstream of one principle. So is logging those agents on your own disk. So is training on what they captured.

TraceLab (UW, June 2026) proved the thesis at scale — 4,265 real coding-agent sessions from 43 developers over eight months. But they anonymize everything for release: tool arguments and results reduced to character counts. Which is the right call for an academic dataset and the wrong call for yourself. The content is the training signal.

So the question I started with was: how do I build a local version of that, keep the content, and make it feed a fine-tune on my own hardware?

The pivot I didn't expect

My first-draft architecture was a LiteLLM proxy, a self-hosted Langfuse, and a raw-JSONL sidecar for export fidelity. I spent a research day on it. The findings weren't encouraging:

I was two days from scaffolding a stack that was heavy and had known fidelity gaps. Then I found liaohch3/claude-tap and the shape of the project changed completely.

What claude-tap does

claude-tap is MIT, Python ≥3.11, about 3,300 stars, 30 contributors, 398 commits YTD. It's a hand-rolled aiohttp proxy — no mitmproxy dep, no openssl, no node — that already does what the first-pass architecture was going to spend a week on:

Which left me with one honest question: what's actually missing?

Three things:

So that's what I built. About 800 lines of Python, Apache-2.0, claude-tap==0.1.145 pinned in pyproject.toml. No docker. No Postgres. No ClickHouse.

The research pass was the valuable part. The code it saved me from writing was almost as valuable as the code I actually shipped.

The architecture, in one paragraph

Every harness points at 127.0.0.1:<port> via ANTHROPIC_BASE_URL (or whichever provider env var is relevant). The claude-tap proxy — one process on loopback — writes each request/response into SQLite with content-addressed dedup. My exporter reads from that SQLite, filters out auxiliary calls (Claude Code's background haiku classifier chatter), flattens Anthropic's content-block structure into OpenAI's messages + tools shape, and writes JSONL. One of those lines, fed to Unsloth with Qwen 3 8B, is a fine-tune.

That's it. There's no diagram worth drawing beyond that one paragraph.

Capturing every session on the host

I run coding agents three ways. Interactive Claude Code at the terminal. vibr8 spawning Claude Code as an ACP backend. Occasionally direct SDK calls from a script. The capture layer has to work for all three without me thinking about which path I'm on.

The clean pattern is claude-tap's proxy-only mode:

# start once, forever
claude-tap --tap-no-launch --tap-port 8080 \
  --tap-store-stream-events --tap-no-live \
  --tap-max-traces 0 --tap-host 127.0.0.1

# set in your shell profile
export ANTHROPIC_BASE_URL=http://127.0.0.1:8080
export CLOUDTAP_DB=/path/to/your/archive/traces.sqlite3

Every subsequent claude invocation — direct, scripted, or spawned by vibr8's ACP dispatcher — reads the env var at startup and routes through the shared proxy. One SQLite, all sessions, no binary wrapping, no $PATH games, no recursion traps. The vibr8 orchestrator doesn't need to know this is happening; the agent subprocesses it spawns inherit the environment.

OAuth stays transparent. Claude Code reads your ~/.claude/.credentials.json, formats the request, hands it to the proxy, the proxy forwards it to api.anthropic.com, bytes come back. The Authorization header is stripped before claude-tap writes the record to disk — I verified this against a real capture, the field value is 15 characters long (just the Bearer sk-ant prefix). Your bearer token never lands on disk.

Does the archive actually preserve fidelity?

I ran a smoke test. One Claude Code session, three sequential tool calls (grep, read, text summary). Then I opened the resulting SQLite with my own exporter path and inspected every claim claude-tap's source made.

What survived unchanged:

Nothing that mattered survived in a weakened form. The archive is the full-fidelity wire record, kept in the agent's native shape — Anthropic traces stay Anthropic, OpenAI traces stay OpenAI. Nothing is flattened at archive time.

Browse the archive in your browser

The archive is a SQLite file, so you can of course go at it with sqlite3 or any client. But claude-tap also ships a built-in browser dashboard — sessions list, per-request detail, tool_use / tool_result pairs, structured diff across adjacent requests, token usage. When you run claude-tap interactively against a client, the viewer is on by default. When you just want to browse an existing archive without launching anything, point CLOUDTAP_DB at the file and run:

CLOUDTAP_DB=/path/to/your/archive/traces.sqlite3 claude-tap dashboard
# ...browse at the URL it prints...
claude-tap dashboard stop

claude-tap dashboard — sessions list for a captured Claude Code trace

That's a one-session view of a throwaway grep + read + summarize run against a two-file scratch directory — same shape you'd see for a real session, just with nothing in it worth hiding.

If capture is all you want, you can stop here

Everything up to this point is the universal story: an on-host proxy per wire protocol (claude-tap for Anthropic Messages, an OpenAI-compatible rewrite for the rest), lossless agent-specific storage, auth stripped, raw SSE events kept for audit. That's the agent-sovereignty answer on its own. "Where did my activity go?" is answered. The archive is yours, on your disk, in exactly the format the agent sent and received.

For a lot of people, that's probably where this stops being interesting. You have your receipts. You can grep them, diff them, hand them to a different model next quarter, feed them into any analysis tool that reads JSON.

The rest of this post is about a second question I happened to care about: can I turn the archive into training data for a model that behaves a little more like what I'd choose, and less like a generic one? That's a strictly harder problem. It opens up curation, normalization, evaluation, and a lot of open research questions. The pipeline I built for it is specific to the Claude Code / Anthropic path — the OpenAI-side normalization is left as an exercise. If you're not planning to fine-tune, the next few sections are optional.

From archive to training data (Claude Code only)

Anthropic's wire format carries a response as a list of content blocks: [thinking?, text?, tool_use, tool_use, ...]. OpenAI's fine-tuning JSONL expects one assistant message with content: "..." and a tool_calls: [...] array, plus separate role: "tool" messages carrying tool_call_id for results. The conversion isn't exciting but it's where fine-tune corpora usually get subtly broken, so it's worth being explicit:

Why OpenAI-messages JSONL and not ShareGPT

OpenAI-messages JSONL is the de facto input format for supervised fine-tuning across every open trainer worth using — Unsloth, axolotl, LLaMA-Factory, TRL. It's also the only widely-supported format with a structural representation of tool calls, as opposed to tool calls serialized as strings inside assistant content. Coding-agent traces are mostly tool call by volume; if the trainer doesn't see them as tool calls, the model doesn't learn them as tool calls.

ShareGPT was never designed for tool calling. The Hermes/Nous convention embeds <tool_call>{...}</tool_call> as literal text in assistant content. LLaMA-Factory extends ShareGPT with a top-level tools field and function_call / observation roles. There is no single structural tool-call representation in ShareGPT — downstream trainers see strings, not calls. axolotl's own docs recommend OpenAI Messages format precisely for this reason. Alpaca is a non-starter for anything with tool use. ChatML is a rendered output of a model's Jinja template, not a storage format.

At training time, the tokenizer's apply_chat_template(..., tools=[...]) handles the model-specific rendering — you ship the structural JSONL and the trainer does the right thing per model.

What this enables

The pipeline exists, which means I now have the raw material for a few things that haven't been possible before on my own hardware:

This piece is about getting the pipeline right. The fine-tune experiment itself is a separate problem for a separate day, with its own open questions I'd rather sit with than rush.

Reproducing on your box

Three commands:

git clone https://github.com/johnrobinsn/trace-forge
cd trace-forge && uv sync

# capture a session:
uv run trace-forge run claude

# export to OpenAI SFT JSONL:
uv run trace-forge sessions
uv run trace-forge export --session-id <id> -o out.jsonl

The repo README has the per-harness setup table (Claude Code, Codex, OpenCode, Hermes, Grok Build, local vLLM via --target http://localhost:8000), the privacy table (what gets captured by default, what the redaction pass catches, how the per-project toggle works), and an Unsloth notebook stub for anyone who wants to try fine-tuning on top of the archive.

Apache-2.0. 800 lines. claude-tap==0.1.145 pinned. No docker.

Honest limitations

Credits

The capture layer is liaohch3/claude-tap's work — the honest version of this post is that the author of that project did the hard part and I built a shim on top. If you build anything in this space, start from their code, not mine.

TraceLab is the prior art that validated keeping agent traces at scale. The paper is worth reading if you care about personal data ownership in LLM-adjacent tooling.

axolotl and Unsloth are the fine-tune stacks downstream of this pipeline. Neither of them needed to care that my dataset exists — the OpenAI-messages JSONL shape is just one of their native inputs.

And vibr8 is why the whole thing exists. Running my own agents on my own hardware for my own work, with the receipts kept on my own disk — that's the whole point.

Repo: trace-forge.

This probably isn't the best way to do this (the fine-tune side)

The capture half is solid — claude-tap is doing the hard work, and the SQLite archive is lossless. If I had to redo that, I'd barely change it.

The fine-tune half is where I'm least confident, and where I'd most like to hear what other people are trying:

If you're working on any of this — a curation layer, a signal-selection heuristic, an evaluation setup that doesn't require hand-grading, a schema that beats OpenAI-messages JSONL for personal-fine-tune fidelity, anything — I'd really like to see it. Reach out, write it up, or just reply to the X thread.

I know this post makes it sound settled. It isn't. The capture half is. The fine-tune half is a direction I'm walking, not a position I'm defending. The next version is almost certainly better, and I'd love to hear about how you're approaching these problems.


Share on Twitter |  Discuss on Twitter

John Robinson © 2022-2026