Surfex

The problem

When you build with an LLM, you may have seen it:

• work from a detailed spec, say it is complete, and still miss items;
• follow a spec change in some places and miss the code elsewhere that depended on it;
• write tests that pass but don't test what the spec requires.

Surfex turns each of these into a failing check.

How it works

Surfex reads every section of the spec, every test and every public function. It keeps a committed, append-only log of which versions of them were shown to belong together.

The key rule is that code is never accepted on anyone's word, a person's or an LLM's. Code counts as implementing part of the spec when a test for that part failed and then passed against it, or when a reviewed test passes against it. A suggestion validates nothing, and no command marks code as implemented without that evidence.

When the spec, a test or the code changes, every relation it touched needs showing again. Surfex says exactly what changed and why. A fresh clone knows what has and has not been validated. CI fails on anything that has not, and an agent gets a precise work list.

Content hashes ignore layout. Moving code, reformatting it or reflowing a paragraph changes nothing. A function's hash covers the private helpers it calls, so a change deep in a helper still counts.

It also handles:

• Test-first work by agents. The spec unit comes first, then a failing test, then the code. Each step is recorded as it happens.
• Adopting an established project. A long-kept test suite can be taken on trust once and earn evidence over time, or every test can be re-evaluated from scratch.
• A spec that is wrong in use. When the tests and code agree with the spec but the result is wrong, the spec itself is marked for change.
• Coverage you can hold a line on. It reports how much of the spec, tests and code is covered by validated relations, and what each missing item lacks.
• Mergeable history. The log merges across branches and flags parallel disagreements.
• Committed reports. Relation status and coverage are kept as files that CI checks for drift.

Getting started

Add it as a dev and test dependency:

def deps do
  [
    {:surfex, "~> 0.10", only: [:dev, :test], runtime: false}
  ]
end

Tell it where the spec and the tests are, and record test runs as evidence:

# .surfex.exs
[
  sources: ["spec.md"],
  tests: ["test/**/*_test.exs"],
  require: [code: [:implements], test_hint: [:verifies]]
]

# test/test_helper.exs
ExUnit.start(formatters: [ExUnit.CLIFormatter, Surfex.ExUnitFormatter])

The workflow then goes one spec unit at a time. A section of the spec carries a short test hint. A test is written from it, tagged with the hint's name, and run until it fails. mix surfex.suggest --accept records the test relation. The code is written until the test passes. Then:

mix test && mix surfex.confirm --evidence
mix surfex.status

The red run and the green run together are the validation. If someone later rewrites the function without changing its behaviour, the same two commands re-validate it. If the spec changes, the test has to change first and fail again.

mix surfex.info lists every topic and command in the installed version. The guides cover writing specs that Surfex can relate well, and adopting an existing test suite.

Status

Experimental. It has helped keep spec, tests and code aligned during LLM-assisted development of several substantial projects. It is under active development, with frequent releases in the 0.x line. Surfex holds its own specification to its own code with its own relation log, and its CI runs mix surfex.status. It has no runtime dependencies and is MIT licensed.