Most model testing starts in a chat window. That works for a quick prompt, but it breaks down as soon as an investigation becomes interesting.
You try several variants. One response calls a tool. Another refuses halfway through. A third behaves differently after a small system-prompt change. Before long, the useful evidence is scattered across tabs, copied JSON, terminal history, and notes that no longer say exactly which configuration produced which result.
I built Lathe to make that work inspectable and reproducible without taking the operator out of the loop.
Lathe is a source-run, local web workbench for manual AI red teaming. It keeps conversations as immutable trees, freezes the configuration behind every model run, streams the model's output and reasoning when the provider exposes it, and records the evidence needed to understand what happened afterward.
It is deliberately not an autonomous scanner. The operator chooses the objective, prompts, models, branches, tools, targets, and approvals. Lathe handles the state, execution boundaries, and evidence.
A session in Lathe: the conversation tree is on the left, the selected branch is in the center, and configuration and run evidence stay available on the right.
A conversation is a tree, not a disposable transcript
The conversation graph is the central idea in Lathe. Messages are immutable nodes, while named branches point to different paths through them. From any turn, an operator can fork a new path, jump between branches, rewind, create a checkpoint, or compare two outcomes side by side.
This makes common testing workflows much less fragile:
- retry a payload without destroying the original response;
- change one model or prompt setting and compare the result against the same history;
- preserve a refusal, partial response, or failed tool call as evidence;
- return to a known checkpoint after exploring several variations;
- export one complete branch in the selected provider's native request format.
Every target-model run captures an exact configuration snapshot. The transcript alone is not treated as enough: Lathe also retains normalized output, usage and timing data, provider stop and refusal signals, tool evidence, and the raw transport trace under the selected redaction policy.
That distinction matters when a provider streams reasoning or partial text and then blocks the generation. Lathe keeps the content that arrived, records the block, and marks the turn visibly instead of flattening the whole event into an empty error.
Bring your own model endpoints
Lathe currently speaks three provider protocols:
- OpenAI Responses;
- OpenAI Chat Completions;
- Anthropic Messages.
Compatible gateways such as OpenRouter work by selecting the protocol their endpoint actually implements, regardless of which company produced the underlying model. Provider profiles are immutable revisions, so changing an endpoint, model catalog, reasoning option, or custom request body does not silently rewrite the setup behind an older run.
The session configuration can also pin ordered system-prompt blocks, tool schemas, output limits, temperature, protocol-specific overrides, and automatic tool-continuation limits. Reasoning, refusal, fallback, and tool-call events stream into the UI as the run progresses.
Real tools, explicit boundaries
Model-visible tools are split into separate pieces: a JSON tool specification, an implementation, and an execution target. A real command can run on the Lathe host, in an existing Docker or Podman container, or over SSH. Tools can also be backed by MCP servers, deterministic mocks, or manual operator-supplied results.
Tool definitions, implementations, targets, and approval policy are selected as exact revisions in the session draft.
Manual approval is the default. An operator can inspect the effective command, arguments, launcher, target, and environment names before allowing execution. Bypass approval exists for deliberately constrained sessions, but it is an explicit, recorded choice rather than a hidden convenience.
Tool failures are also part of the conversation. A permission error, missing file, nonzero exit, timeout, or MCP error is returned to the model as an error result so it can recover or explain what happened. The failure remains visible in the graph and evidence; it is not relabeled as success merely because the model is allowed to continue.
A workbench for developing the next payload
Some red-team prompts are easier to build as artifacts than to type directly into a chat box. Lathe's Payload Workbench provides three related workflows:
- Transform applies deterministic encodings, text operations, variable rendering, and saved pipelines while preserving each intermediate revision.
- Generate asks a separately configured helper model for one or more candidates using selected instructions, techniques, variables, and a deterministic context budget.
- History retains generations, refinements, edits, transformations, traces, and provenance so a previous candidate can be restored or compared exactly.
Helper generation is detached from the target conversation. Nothing enters the graph until the operator explicitly chooses “Use as next prompt.”
Generator profiles can reuse an existing HTTP provider revision or a locally installed Codex App Server with the operator's existing ChatGPT login. Multiple candidates are independent calls, partial results remain inspectable, and refinement creates a child revision instead of overwriting its source.
Context is intentional rather than implicit. The operator can include project and session briefings, choose no/minimal/full conversation context, optionally attach the current target configuration, and inspect the exact compiled preview before generation. Minimal mode preserves reasoning and complete tool calls while truncating large tool results deterministically.
Evidence that can leave the app
Useful results can be recorded as findings and exported with their reachable transcript, configuration snapshots, run evidence, and accepted payload lineage. Harnesses, findings, and branch request JSON serve different purposes:
- a harness packages a reusable testing configuration;
- a finding packages a reproducible observed result and its evidence;
- a branch export produces the provider-native request body for the selected root-to-head path.
Imports remain cautious: executable assets arrive untrusted, identifiers are remapped, and credentials and local authentication state are excluded. Lathe also keeps provider profiles and library assets revisioned so older evidence continues to point at the exact material that produced it.
Getting started
Lathe currently targets macOS and Linux with Node.js 24 and pnpm 11:
$ git clone https://github.com/rekter0/lathe
$ cd lathe
$ corepack enable
$ pnpm install --frozen-lockfile
$ pnpm devThe server prints a tokenized loopback URL. Open that exact URL, add a provider, create a project and session, select a harness or assemble the prompt and tool configuration, and start a branch.
Lathe is early v1 software, but the direction is already clear: model red teaming should feel less like juggling chat tabs and more like working in an instrumented lab. The operator should remain in control, while every branch, configuration change, tool outcome, provider block, and promising payload stays available for inspection and reproduction.