Projects / HAI
HAI
A cheap coding agent that learns from its own verified work, aiming for near-frontier results on a codebase at a fraction of the cost. Plus a public record of whether it works.
The question
Training intelligence is expensive. Using it doesn't have to be. Can a cheap, capable model get close to a frontier model on a specific codebase, at a fraction of the cost, by remembering what worked, looking things up and checking them against reality, and improving how it approaches problems?
The idea is a veteran effect. Someone who has worked in a codebase for a year knows where things live, which conventions are real and what broke last time. A general model arrives knowing none of that. HAI tries to give a small model that experience, and to keep it honest: its memory is mostly code (tests, probes and checks it can run before trusting them), and anything that fails is retired.
The claim being tested is cost per solved task at near-frontier accuracy. It isn't a claim to beat the best models in general. Search and memory add knowledge, not reasoning, so the first thing to find out is how many failures are really missing knowledge.
How it's built
Four layers. Only the harness stays fixed; everything above it is allowed to change, but only through a gate.
| Layer | What it is | How it changes |
|---|---|---|
| Harness | fx, Vercel Labs' small, embeddable coding agent | Doesn't. Kept fixed so results are comparable. |
| Model | DeepSeek V4.1 Flash through the OpenCode API; a small local model later for weight updates | Swapped only between experiments, never mid-run. |
| Memory | Lessons from past runs plus runnable checks about each codebase, in SQLite behind an MCP server | After each task, but only from runs the tests judged. Items that start failing are retired. |
| Strategy | Prompts, skills and small tools, versioned in git | An evolution agent proposes changes; they're kept only if they score better on a validation set with no regressions. |
The plan
Each stage is measured against the one before it. If a stage doesn't help, that's a result, and it's recorded.
- Baselines. fx with bare DeepSeek V4.1 Flash, and one frontier model as a reference, on the same tasks. Sets the starting score, the cost per solved task, and how noisy runs are.
In progress - Triage and oracle test. Sort the baseline's failures into missing knowledge versus weak reasoning, then give the model the right hints on those tasks. This sets the ceiling for everything after it, and can end the idea early.
- Notes, memory and search. Compare a plain notes file, HAI memory, and search with verification, each against bare Flash. HAI memory has to beat the notes file, not just nothing.
- Executable memory. Add tests and probes the agent can run before trusting a lesson. Does runnable memory beat prose?
- Self-improving strategy. Let the agent propose changes to its own prompts and tools, keeping only those that win without breaking anything.
- Weight updates. Fine-tune a small local model on its own verified wins, and compare against memory alone.
How progress is measured
Tasks run as an ordered stream per codebase, so the agent can learn as it goes. Every 25 tasks, it's scored on a separate held-out set with memory frozen. Improvement there is learning; improvement only on tasks it has already seen could just be memorising.
- Cost per solved task: the full spend, including retrieval, search, verification and retries, divided by tasks solved. The headline number, shown against accuracy.
- Held-out pass rate: share of unseen tasks solved.
- Regression rate: tasks that used to pass and now fail. The main check on self-improvement.
- Steps per solved task: whether it gets more efficient, not just more accurate.
- Memory size against use: to catch the store bloating with things it never needs.
Tasks come from public benchmarks (a Terminal-Bench and SWE-bench Pro subset) and from real repositories turned into tasks with tests. Public repos prove the method; a private or very recent repository is the cleaner proof, because the model can't have seen it in training.
Stopping rule, set in advance: if memory doesn't add at least 3–5 points on held-out tasks after about 200 tasks, or doesn't beat the plain notes file, the main idea is weak, and the layers above it won't rescue it.
Built on
- fx for the harness
- ReasoningBank for the shape of memory items
- A Self-Improving Coding Agent (SICA) for the strategy loop
- Code as Agent Harness, a survey that frames the whole approach