Search-Based Specification Hierarchy Recovery: Reverse-Engineering Codebases into Optimized Specification Trees with SpecTreeGenerator
Lars Shakya Buch-Jepsen
Introduction
Specification trees—hierarchies of specification (spec) files describing a system’s components—are valuable for program comprehension, onboarding, and audits, yet they almost never exist in sync with the code they describe. Recovering the links between source code and free-text documentation is a long-standing traceability problem [1], and the value of the recovered material depends heavily on how it is structured; the principles of sound decomposition go back to Parnas [2].
SpecTreeGenerator, a .NET 8 tool from DevelApp-ai, addresses this by constructing the specification structure automatically: it reverse-engineers a codebase into a tree of spec files, enriches the tree with documentation extracted from Markdown, HTML, Word, and plain-text sources, and then applies a genetic algorithm to optimize the tree itself—removing redundant content and pushing shared content into inherited sections, so that each node states only what distinguishes it from its ancestors.
This article describes the shipped architecture as verifiable in the repository, positions the work in the research literature, and situates the tool within the SpecTree ecosystem that consumes its output (background section). It deliberately reports no quantitative results; the required measurements are enumerated in an evaluation-status section, and the measurement harness that will produce them now ships with the tool.
Background and Related Work
SpecTree: The Specification-Tree Artifact and Its Ecosystem
Within DevelApp-ai, the specification tree this tool recovers is a named artifact: SpecTree, a living tree of specification files co-located with the code, defined as the single meeting point between humans and code agents in the development process [3]. A SpecTree mirrors the system decomposition. Specification content is composed by concatenation from the root down to each node, so a node’s effective context is its ancestry; references point only upward, so circular dependencies between specifications are impossible by construction rather than by review vigilance; and intent is authored as natural-language prompt specification files paired with test specifications, from which code agents implement under a least-change rule. SpecTree is the evolved form of the organization’s specification-driven methodology, and its defining commitment is that every crossing of the human-agent boundary is a versioned, reviewable Git event.
SpecTreeGenerator holds a defined seat in this ecosystem: it is the bootstrapping handover, from existing code to a first specification tree, so that a system that was never specification-first can still gain one. Its machine-readable output contract, spectree.json, is the shared format that downstream tooling consumes (described in the System Overview). The consuming side is deliberately out of scope here and is treated in companion work:
- the deterministic Technical Design Specification (TDS) pipeline, the subject of the companion article in this collection, addresses the forward path of the same problem: design structure is authored up front as data (machine-readable decision records, requirements, and policy metadata), validated deterministically, and merge-blocked when a change skips the governing decision, with recovered spec trees ingested back in as first-class declaration artifacts [4];
- specification-first grammars make the structured documents a tree carries (solution manifests, software requirements, architecture decision records, APIs, models, and tests) parseable and validatable artifacts, complementing the natural-language prompt files;
- collaborative-modeling and citizen-development surfaces round-trip spec trees through the shared contract: event-storming canvas sessions export as per-microservice specification folders, and a citizen interface lets domain specialists edit, validate, and submit specification changes for review and merge.
All of the above is shipped functionality in the organization’s repositories. The method-level treatment of SpecTree is the field-guide chapter cited above; the deterministic governance side is the companion TDS article; and this article covers the recovery tool that feeds both.
Traceability Recovery
Antoniol et al. formulated traceability recovery between code and documentation as an information-retrieval problem and established the probabilistic and IR baselines on which later work built [1]. SpecTreeGenerator differs in goal: rather than recovering links between pre-existing artifacts, it generates a specification hierarchy whose shape is itself optimized.
Search-Based Software Engineering
Search-based software engineering reformulates software-engineering problems as search problems solved with metaheuristics [5], [6]. The closest relative to SpecTreeGenerator’s optimizer is Bunch, which uses hill climbing to partition a system into near-optimal module clusters for remodularization [7]. Bunch optimizes dependency-driven clustering objectives; SpecTreeGenerator instead optimizes a content-driven documentation objective: minimizing textual redundancy across a spec tree. To the authors’ knowledge, applying genetic optimization to the shape of a generated specification tree is a novel framing, and no evaluation yet exists to substantiate it.
Documentation Analysis
Mapping textual artifacts onto code entities is commonly approached with topic models such as latent Dirichlet allocation [8]. SpecTreeGenerator’s documentation extraction performs an explicit, deterministic mapping from documents to code components rather than a statistical one, favoring precision over recall.
System Overview
The following components ship in the repository (DevelApp-ai/SpecTreeGenerator):
- Roslyn analysis backend. For C# code, the tool builds a full Roslyn semantic model—symbols, inheritance relationships, and member structure—providing the highest-fidelity view of the analyzed system.
- Minotaur analysis backend. For languages other than C#, analysis falls back to the Minotaur platform: grammar-based analysis where a Minotaur grammar exists for the language, and coarser structural scans otherwise. The tiered fidelity is a deliberate design trade-off; per-language fidelity has not been measured.
- Documentation extraction. Markdown, HTML, Word, and plain-text sources are normalized to Markdown and mapped onto the code components they describe, becoming the prose content of the spec tree.
- Machine-readable output contract. The generated tree is exportable as spectree.json, a documented machine-readable schema with a CLI for embedding the tree in other tooling.
- Optional AI semantic analysis. Where configured, Mistral models via Microsoft Semantic Kernel semantically enrich the extracted specifications. A deterministic offline fallback runs when no model is reachable, keeping runs reproducible without network access.
- Genetic-algorithm optimizer. The hierarchy produced by the extraction pipeline is restructured by a genetic algorithm [9], described next.
Genetic Optimization of the Spec Hierarchy
The chromosome is the spec tree itself: an arrangement of spec content into a rooted hierarchy. Fitness rewards hierarchies in which content shared by several nodes is stated once in a common ancestor (through inheritance) and penalizes duplicated content across sibling subtrees. Standard genetic operators adapted to trees—subtree crossover and mutation of node placement—drive the search, in the tradition of genetic programming [10].
Conceptually, this is the same move Bunch made for module clustering [7]: reformulate a comprehension-support structure as an optimization problem. The objective differs: Bunch optimizes coupling and cohesion of dependency clusters, whereas SpecTreeGenerator optimizes non-redundancy of the generated specification text.
The optimizer is implemented. Its parameter settings, convergence behavior, and the quality of its output relative to simpler baselines (for example, a flat or alphabetical spec layout) are unmeasured. All claims in this section describe design intent, not results.
Discussion
The architecture has three deliberate strengths: precision where it matters most (Roslyn for C#), graceful degradation rather than hard failure for other languages (the Minotaur backend’s tiers), and reproducibility by default (the deterministic offline fallback). Its main limitations mirror these choices: fidelity outside C# varies by language and is currently unquantified; the mapping from documentation files to code components relies on heuristics whose precision is unmeasured; and genetic fitness is a proxy for human usefulness, not a measurement of it.
Evaluation Status
Evaluation status. This article reports shipped functionality only. No quantitative result appears in it yet. The measurement infrastructure now ships with the tool: the spectree-measure CLI (hierarchy, optimizers, sensitivity, docmapping, and languages commands) runs the real pipeline deterministically under seed control, and a CI workflow executes the measurement battery on real repositories and uploads raw JSON results with a manifest for reproducibility; the companion issue that enumerated the measurements was closed by this harness. The measurements themselves—(1) hierarchy quality versus baselines, measured as reduction of duplicated spec content; (2) genetic-optimizer convergence and parameter sensitivity; (3) precision of the documentation-to-component mapping on a hand-labeled sample; (4) runtime and memory across repository sizes; and (5) a small developer study on comprehension—have not yet been committed as data, and until they are, no number belongs in this article.
Conclusion
SpecTreeGenerator combines precise static analysis, multi-format documentation extraction, and search-based restructuring of the result into a tool whose output—a redundancy-minimized specification tree—is a plausible novel object of study in search-based software engineering. Whether it is a good one is exactly what the pending measurements must establish.