Skip to content

Architecture overview

This section is for developers and architects. It does not assume radio astronomy knowledge; the glossary defines domain terms. Astronomers looking for the algorithms should read How Hebog finds sources.

What Hebog is

Hebog is a Python library with one public operation:

result = hebog.find_sources(request, config, executor)

It reads one astronomical image (a FITS file), finds the objects in it, and writes a catalogue, a noise image, a mask and a diagnostics file. It is designed to be embedded in a larger data-processing pipeline that already owns a Dask cluster. Its first intended consumer is Rapthor, a LOFAR imaging pipeline, where it replaces PyBDSF in one step. The adapter that runs one Rapthor sector and writes Rapthor's files, hebog.adapters.find_sector_sources, is implemented; Rapthor's own task that calls it is not yet.

Quality goals

Priority Goal Consequence
1 Scientific equivalence with what the consuming pipeline needs Serial execution is the reference; every other mode must reproduce it
2 Scalability to 100,000 × 100,000-pixel images on hundreds of nodes No step may need a whole image on one worker
3 Performance: the source-finding part of the pipeline step at least twice as fast as the PyBDSF reference where PyBDSF is slow, on sectors with many sources, and no slower elsewhere Measured end to end, never from isolated kernels
4 Interoperability with any pipeline No dependency on Rapthor, Prefect or LSMTool; Dask is imported only when a Dask executor is requested
5 Maintainability Pure functions, immutable records, inward-pointing dependencies

Quality attributes and coding principles gives the rationale and the enforced gates.

System context

flowchart LR
    pipeline["Calling pipeline<br/>owns workflow, retries, cluster"]
    hebog["Hebog<br/>find_sources()"]
    input[("FITS image")]
    products[("Catalogue, RMS image,<br/>mask, diagnostics")]
    dask["Dask cluster<br/>(optional, caller-owned)"]

    pipeline -->|"request, config, executor"| hebog
    input --> hebog
    hebog -->|"coarse tasks"| dask
    hebog --> products
    hebog -->|"small result record"| pipeline

Hebog analyses one image per call. Scheduling across images, retries, resource budgets, and any translation into another tool's file conventions belong to the caller. The domain model shows the wider Rapthor context.

Layers

Dependencies point inward. An inner layer never imports an outer one.

flowchart TD
    adapters["adapters/<br/>Rapthor-compatible records and catalogue view"]
    api["pipeline.py · public_api.py<br/>validation, admission, I/O, atomic publication"]
    public_science["public_science.py<br/>terminal catalogues from the published records"]
    stages["stages/<br/>the stage sequence; tiling, halos, batching through an Executor"]
    science["science/<br/>reviewed profile and configuration, records, catalogue-row kernels"]
    algorithms["algorithms/<br/>pure NumPy/SciPy kernels"]
    shared["data_models/ · config.py<br/>immutable records shared by every layer"]
    executors["executors/<br/>Serial · Thread · Dask"]
    io["io/<br/>FITS input, Zarr planes, product files"]

    adapters --> api --> stages --> science --> algorithms
    api --> public_science --> science
    api --> io
    api --> executors
    stages --> executors
    stages --> io
    science -.-> shared
    stages -.-> shared
    algorithms -.-> shared
Layer Responsibility Must not
algorithms/ Arrays and immutable configuration in, arrays or records out know about schedulers, files or adapters
science/ The reviewed profile and configuration, the composition records and the catalogue-row kernels import stages, executors, io or adapters
stages/ Apply the kernels of algorithms/ and science/ over tiles, halos and coarse batches; composition.py runs the stages in order hold scheduler-specific objects
public_science.py Build the terminal catalogues from the records the stages published run stages or read planes
public_api.py Validate and admit the input, run the stage sequence, publish products atomically leak open files or arrays into results
executors/ Run batches serially, on threads, or on a caller-owned Dask client create clusters
io/ FITS windows in, Zarr intermediate planes, FITS and JSON products out —
adapters/ Translate to a consumer's names and formats be imported by inner layers

LAYER_IMPORTS in tests/unit/test_architecture.py states, as one table, which layers each layer may import. An import outside it needs a named exemption with its reason, and the test fails when an exemption stops matching. The same test keeps Rapthor, Prefect and LSMTool out of the package, adapters included, and Dask inside executors/, apart from one named exemption: the execution profiler's local cluster in hebog.validation, which wheels exclude. ADR-009 records why science/ sits below stages/.

Importing hebog performs no I/O and does not import Dask; DaskExecutor loads only when requested. hebog.validation is development tooling and is excluded from wheels; no production layer imports it.

Key decisions

Decision Record
The caller owns scheduling; Hebog accepts an executor ADR-004
Large images are split into tiles with halos and reconciled hierarchically ADR-005, ADR-008
Zarr is the only intermediate image store; FITS is for input and final products ADR-007
Internal schemas are versioned; compatibility formats live in adapters ADR-006
The reviewed science sits below the stages that apply it ADR-009
Scope is limited to what the first consumer needs ADR-003
Native code only after measured gates Native-code assessment

All records are in the ADR index.

Where to go next