Depdep

The problem

LLM-assisted development produces many small changes, and each one runs a pipeline. Every pipeline that recompiles its dependencies spends time before any test runs. Two cases come up often:

• Several Mix projects in one repository (a "poncho") compile the same package once per member. The repository depdep was extracted from had 564 dependency instances across 113 distinct packages, with one large package compiled ten times at 44 seconds a pass.
• A single project recompiles its dependencies on every pipeline.

A CI cache usually handles the second case. A cache restores an opaque archive, then leaves Mix to decide what is stale by comparing file timestamps. That steps around the machinery Mix uses to stay correct, and the failure is silent. The build compiles clean, passes its tests, and could be wrong, and nothing tells you either way.

How it works

Depdep keys each compiled dependency by a hash of everything that went into it: the toolchain, the dependency's own source, the build options it was declared with, the compile-time configuration reaching it, and, recursively, the keys of its own dependencies. So a stored object either matches the build you would have produced yourself or is not returned at all.

Compile-time configuration matters because values from a consumer project's config can be baked into a dependency's bytecode. Two projects that configure the same package differently need different objects, and depdep's key tells them apart.

Debian packages and git mirrors are stored too. Each is keyed in the way that is sound for it: a .deb needs nothing beyond its own filename.

Other design choices:

• No runtime dependencies. Depdep runs before mix deps.get, so it cannot depend on what that step fetches.
• A miss is not an error. With no store configured, depdep says so and exits cleanly, and Mix compiles as usual. A pipeline can be wired up before credentials exist.
• Least privilege. Depdep needs only get and put on one bucket of an S3-compatible store. Clearing out old objects is a separate, narrowly scoped command that removes nothing unless explicitly confirmed.

Getting started

Depdep runs before dependencies are fetched, so it is not listed in your mix.exs. Instead you commit a small script that installs it:

# scripts/depdep.exs
Mix.install([{:depdep, "~> 0.12"}])

Depdep.CLI.main(System.argv())

You point it at an S3-compatible store with environment variables and run it as the first step of a CI job. The getting-started guide covers the store, CI wiring and how to check that restores are working. Separate guides cover poncho projects, apt packages and git mirrors, store cleanup, and the design.

Status

In use. Depdep runs first in the CI of the projects it was built alongside, restoring compiled dependencies and Debian packages on every pipeline. Measured across six of those projects, the net saving per pipeline was positive at every one, from about 8 seconds to about two minutes. On the repository it was extracted from, a cold-cache CI run went from about 28 minutes to about 6.5 minutes. That figure is a ceiling for a first adoption, not what a warm pipeline sees.

The defects found so far have cost time, never correctness. When depdep misjudged a restore, it rebuilt work it could have reused. It has never served a build that did not belong.

It has been run against SeaweedFS and MinIO. Other S3-compatible stores meet its requirements but have not been tested. Depdep is published on Hex and MIT licensed.