ADR-006: Isolate compatibility with versioned internal schemas
| Status | 🟢 Accepted |
| Created | 2026-07-18 |
| Last Updated | 2026-10-08 (the catalogue schema's current version and what the architecture test checks) |
| Deciders | Gemma Danks |
| Tags | compatibility, schemas, Rapthor, PyBDSF, interoperability |
Context
Rapthor currently consumes paths to materialised source-finding products: source catalogues, true-sky and flat-noise RMS images, an island mask, filtered sky models, and diagnostics. It reads a small set of PyBDSF catalogue columns and relies on LSMTool-compatible filtering and empty-result behaviour, but it does not consume live PyBDSF image, island, Gaussian, or source objects.
ADR-003 limits the initially qualified feature set to this contract while keeping Hebog reusable in other pipelines. The Phase 0 inventory identifies legacy names and product suffixes that must be preserved at the Rapthor boundary. Hebog still needs clear internal concepts for sources, islands, measurements, masks, image planes, partitioned products, and materialised results. Large images also require chunk and catalogue-shard records that do not exist in the current PyBDSF product model.
Phase 1 has now defined the initial internal catalogue and materialised-result fields, null representation, ordering, relationship validation, and on-disk FITS and JSON representations. They remain provisional until the Phase 0 human scientific sign-off; the boundary and evolution strategy in this ADR remain unchanged.
Problem Statement
Should Hebog use versioned domain-oriented internal schemas with an isolated compatibility adapter, mirror PyBDSF/LSMTool schemas throughout the package, or translate ad hoc dictionaries only when writing products?
Options Considered
| Option | Description | Scientific clarity | Rapthor compatibility | Other workflows | Evolution safety | Large-image products | Overall score |
|---|---|---|---|---|---|---|---|
| Weight | - | 2 | 2 | 2 | 2 | 1 | - |
| Versioned internal schemas | Typed domain records mapped by outer adapters | ✅ | ✅ | ✅ | ✅ | ✅ | 27 |
| Mirror legacy schemas | Use PyBDSF columns and LSMTool names internally | ❌ | ✅ | ❌ | ⚠️ | ❌ | 14 |
| Ad hoc dictionaries | Translate loosely structured mappings at write time | ❌ | ⚠️ | ⚠️ | ❌ | ⚠️ | 12 |
✅ = 3 (good), ⚠️ = 2 (acceptable), ❌ = 1 (poor)
Decision Outcome
Hebog will use versioned, domain-oriented internal schemas with isolated compatibility and workflow adapters.
Internal records use the agreed Hebog vocabulary and make coordinate frames, units, array axis order, identifiers, nullability, and product roles explicit. They are small, typed, serializable, and free of open files, scheduler clients, workflow state, and image-sized arrays. Image planes remain behind bounded I/O boundaries; schemas refer to paths, logical dataset identifiers, chunks, checksums, windows, or small summaries.
Every materialised or externally exchanged schema has an explicit integer schema version. Readers validate supported versions and reject unknown or ambiguous fields rather than guessing. During pre-production, a semantic or structural breaking change updates the current contract and schema version; stale development artifacts fail clearly and may be recreated. Hebog does not maintain legacy readers, migration code, or compatibility tests before 1.0 unless the user explicitly requests them for a particular interface. Breaking changes remain visible in current documentation and release notes. The choice of dataclass, Pydantic model, FITS table representation, or another implementation mechanism is made per concrete Phase 1 contract; this ADR does not require one schema library everywhere.
The catalogue schema, now version 3, uses strict immutable Pydantic records
and canonical JSON. It distinguishes islands, source candidates, and fitted
Gaussian components; uses ICRS degrees, Jy, Jy/beam, hertz, explicit position
epoch, and an explicit spectral convention; represents unavailable values as
None; and requires stable unique canonical identities. It is MFS-only and
rejects mixed reference frequencies.
SourceFinderResult schema version 2 replaces its earlier path-only scaffold.
Each concrete materialised product now records a role, media type, content
schema, byte count, SHA-256, and scientific status. Existing path properties
remain available to consumers. Only RMS may be scientifically unavailable in
a successful core result, preventing copied input pixels from being described
as an estimate. No implemented Hebog source-finding pipeline emitted result
schema version 1.
Internal final products use Astropy-backed FITS for the catalogue, RMS, and
mask, and canonical JSON for diagnostics. Image materialisation consumes
bounded row blocks and preserves an explicitly selected float dtype for RMS;
the mask is binary uint8. A complete validated file is published from a
same-directory temporary path, and the returned MaterializedProduct binds
it to a byte count and SHA-256. These encodings do not adopt PyBDSF column
names or legacy placeholder rows; the compatibility adapter remains
responsible for those translations.
The Rapthor adapter depends inward on Hebog's public pipeline and schemas. It
owns legacy catalogue names such as Source_id, Isl_Total_flux, and
DC_Maj; required suffixes; PyBDSF/LSMTool unit and null conventions;
true-sky/apparent-sky component matching; mask-based grouping; diagnostics
translation; and reviewed empty/failure compatibility. No scientific kernel
imports those names or external packages.
Other workflows may consume the internal materialised schema directly or provide their own adapter. They do not pass through a Rapthor-shaped product unless they genuinely require that compatibility contract. Frozen PyBDSF products remain validation inputs, not the internal schema definition or a runtime dependency.
Consequences
- Good, because scientific code uses precise domain names instead of legacy column and workflow terminology.
- Good, because Rapthor compatibility can evolve and be tested without coupling every algorithm, executor, and alternate workflow to PyBDSF.
- Good, because explicit schema versions make stale pre-1.0 artifacts fail clearly without obliging the project to maintain migrations or legacy readers.
- Good, because bounded chunk, shard, and partition records can support 100,000-by-100,000 images without pretending they are single legacy files during intermediate stages.
- Bad, because every compatibility product requires an explicit, tested mapping and may require a final materialisation pass.
- Bad, because Hebog must maintain its internal schema and the Rapthor adapter until the legacy boundary is retired.
- Bad, because early adopters may need to update code or recreate development
artifacts after a breaking
0.xchange. - Risk: duplicating large image or catalogue data during translation could erase performance gains. Adapters must stream, map columns, and materialise from bounded chunks or shards rather than copy complete large products in memory.
- Risk: a superficially compatible catalogue can still alter LSMTool or Rapthor decisions. Contract and acceptance tests must validate downstream behaviour, not field names alone.
Confirmation
- Architecture tests reject imports of Rapthor, Prefect and LSMTool from every module, adapters included, and of a concrete scheduler outside the executors except for named exemptions.
- Phase 1 schema tests cover version validation, physical domains, nulls, ordering, referential integrity, empty catalogues, product roles and status, unsupported versions, and deterministic serialization.
- Phase 1 materialisation tests cover populated and zero-row catalogues, bounded RMS and mask writes and reads, explicit unavailable RMS, canonical diagnostics, content identity, idempotent retries, conflicts, corrupt structures, and unsupported content versions.
- Rapthor adapter contract tests cover every consumed catalogue field and product, normal and empty paths, filtered-model membership/grouping, and diagnostics source counts.
- Frozen released-PyBDSF and pinned-
masterproducts are converted through the same adapter used by the comparison runner; tests never redefine mappings locally. - A non-Rapthor workflow smoke test consumes Hebog-format materialised results without importing Rapthor, Prefect, or LSMTool.
- Reviews reject legacy product names in scientific kernels and reject public records containing live external objects or image-sized arrays.
Links
| Type | Links |
|---|---|
| ADRs | ADR-003, ADR-004, ADR-005 |
| Documentation | Rapthor source-finding contract, Domain glossary |
| Plan | Implementation plan |