Skip to content

Capability and status

Hebog is a scalable source finder for radio continuum images. It is experimental: it runs a complete standalone analysis on the inputs below, but is not scientifically qualified for survey use and is not a production Rapthor backend.

This page describes the source checkout, which can be ahead of the latest release. Pin an exact 0.x version when results must be repeatable.

Supported inputs and behaviour

Aspect Supported
Input One two-dimensional FITS image; leading axes of length one are allowed, and a Stokes axis must select Stokes I. The input header contract lists what each common imager writes and what to supply.
Units BUNIT=Jy/beam, or a supplied unit when the header has no BUNIT.
Coordinates ICRS or FK5 J2000 celestial WCS. A header with EQUINOX = 2000 and no RADESYS, as written by WSClean, is FK5 J2000. Catalogue positions are always ICRS. Other frames are rejected.
Beam and frequency Finite positive BMAJ and BMIN, a BPA, and a positive reference frequency. The request can supply a value the header omits, never one it already has. A beam wider than 10 pixels (FWHM) is refused.
Image size At most 22,500 pixels on each side, and at most 1,000,000 pixels in all when the shorter side is under 600. Other images fail before analysis.
Pixel values Any BITPIX. Stored integers are scaled by BSCALE and BZERO.
Invalid pixels NaN pixels, stored integers equal to BLANK, and every pixel of a 3×3 square of one repeated value, such as zero padding, are excluded from estimation, detection and measurement.
Profiles continuum (default), or compact, which omits extended-source association and reports extended-emission-incomplete. The background is estimated (default) or, as PyBDSF's zero mean map, zero.
Thresholds Caller-set detection and island thresholds (island below detection), minimum and optional maximum island size.
Execution SerialExecutor, ThreadExecutor, or DaskExecutor with a client you own. Dask workers need the image and the output directory's parent on shared storage. All must give the same products. On Windows, threads of one process take turns at FFT convolutions, because SciPy's Windows wheels share an unlocked FFT plan cache.
Output A new directory with catalogue.fits, rms.fits, source-mask.fits and diagnostics.json, and flat-noise-rms.fits when the request names the sector's flat-noise image. Existing directories are never overwritten.

Every scientific step runs on bounded tiles through the executor, an image larger than 2,048 pixels on either side is reconciled across several tiles rather than held as one, and the driver holds no image-sized plane. The size limit rises one tier at a time, as each tier's memory and invariance evidence is measured. A serial run of the whole 22,500² LOFAR-HD ELAIS-N1 mosaic allocates at most 1.8 GiB, almost all of it what the analysis holds between tasks, which grows with the image, and keeps about 50 GiB of intermediate planes on disk. Bounding both is planned before much larger images are admitted. A four-worker local Dask cluster finishes the 10,000² LoTSS-DR3 anchor in about 0.4 of the serial time, with identical products. Threads in one process barely speed a run up, because Python's global lock holds them: twelve took 0.84 of one thread's time on that anchor. Run in parallel with Dask workers in separate processes.

Every task declares the memory it holds, so an executor with a memory budget runs only as many at once as fit, and a DaskExecutor given the names of the workers' memory and core resources lets Dask place each task only where that much is free. hebog.analysis_memory_bytes states what the analysis itself holds, which grows with the sources it finds and the image, for the caller to reserve, and hebog.expected_source_count estimates those sources from the image (integrate Hebog into a pipeline). One declared limit remains: an object wider than a task's read budget is reduced on the driver from its own pixels, so that memory grows with the object rather than the tile. A connected object filling a 22,500-pixel field would need about 94 GB on the driver; no object in the real LoTSS-DR3 fields measured comes within a factor of ten of the budget. The performance profile and progress against goals have the dated figures.

Scientific status

Hebog is suitable for demonstrations, integration work, algorithm inspection and bounded evaluation. It is not yet suitable for claiming interchangeability with PyBDSF or readiness for a survey.

Development comparisons against PyBDSF and Aegean on simulated and public images are summarised on progress against goals. They are development evidence, not qualification.

Known limitations when interpreting results:

  • Uncertainties are not yet calibrated across all morphologies and signal-to-noise ratios.
  • Extended-source flux errors have wide tails and centroids can be offset.
  • Faint extended emission is sensitive to association, mask-boundary and aperture decisions.
  • A detection whose Gaussian fit fails has no Gaussian-component row.
  • An image with no usable positive RMS yields an all-NaN RMS product, an empty catalogue and a zero mask. This is not evidence of an empty sky.
  • The background and noise meshes are fixed in pixels, so a beam many pixels wide is not measured well: on injected isolated sources at SNR 10 or more, the continuum profile misses some from 12 to 14 pixels and the compact profile from 16. Beams above 10 pixels are refused; at every beam measured from 3 to 10 pixels every such injected source was published. A pipeline whose images have wider beams needs the meshes scaled with the beam, which is deferred work.
  • The published RMS reads slightly low on pure noise. Each noise window is clipped at three standard deviations and the standard deviation of what remains is published without a correction for the truncation: about 1.6% low on white noise and 3 to 4% low on beam-correlated noise. PyBDSF applies a correction of about 1.4%. Whether to apply one is being measured.
  • In a field so crowded that no fine noise window lies clear of its sources, the background and RMS are the unprotected sigma-clipped coarse estimate, which includes the sources' wings.
  • A source on the wing of a much brighter one that makes no peak of its own gets no component. Its flux stays in the residual, which can join the brighter source and that source's other neighbours into one source. In a generated 1,024-pixel field with a source every 32 pixels, one in ten of them resolved, 995 of 1,024 injected sources have a published Gaussian, where PyBDSF publishes one for 1,005, and the field's 960 islands become 994 sources where PyBDSF finds 1,006.
  • The compact profile is not a general continuum catalogue.
  • Completeness, reliability, astrometry and photometry must be evaluated on data representative of your use.

Always read diagnostics.json with the catalogue. It records every detected source and component, including those without a catalogue row. Its configuration_qualification field reads development-unqualified for the example 5σ/3σ/seven-pixel settings and custom-unqualified otherwise; both describe validation status, not whether the run succeeded.

Integration status

The standalone API is implemented and tested as an independent library. Requests and results hold paths and small serializable records, never open files, image arrays or scheduler clients. Distinct exception types separate invalid input, unsupported metadata, oversized images and existing output directories, and a retried request returns the products its finished run published.

hebog.adapters.find_sector_sources runs one Rapthor sector, its true-sky and flat-noise images, on the executor Rapthor builds, and writes the catalogue, the two RMS maps and the island mask under the names Rapthor's PyBDSF step uses. It runs the continuum profile with a zero background unless asked otherwise: of the four settings, continuum passes the truth-based gate under either background, and the zero background covers more of the true emission and keeps PyBDSF's mean map. Rapthor's Hebog branch calls it from the existing Prefect task with a ThreadExecutor on Rapthor's current one-single-threaded-worker-per-node layout. It includes memory admission, typed PyBDSF fallback and dual-run diagnostics; it is not yet merged. Its distributed path is future work. When no beam Measurement Set is supplied, the branch passes the flat image as both inputs, matching LSMTool's detection and RMS convention. Hebog does not:

  • measure sources independently on the optional flat-noise input, of which it publishes only the RMS map;
  • filter or group a sky model, which LSMTool's filter_sources does with the island mask;
  • manage retries, scheduling or cluster resources; or
  • provide a qualified replacement for Rapthor's complete filter_skymodel step.

See Integrate Hebog into a pipeline.

Compatibility between releases

Hebog promises no backward compatibility between 0.x releases. An API, output schema or measurement can change directly. Products carry schema versions, and Hebog's readers reject unsupported versions and verify each file's role, size and SHA-256. Schema numbers are listed in the output reference; they are software checks, not scientific maturity levels.

hebog.validation is development tooling for this repository's comparison scripts. It is not part of the public API and is not installed from a wheel.

Release boundaries

A release has a tested public workflow, an installable package, current documentation and stated limitations. It does not imply scientific equivalence with another finder, qualification for a survey, support beyond the table above, Rapthor integration, a demonstrated speed advantage, or compatibility with later releases.

Releases are tagged on GitHub and, from 0.22.0, published to PyPI, so that Rapthor's optional Hebog dependency installs normally. Being on PyPI changes none of the above: every 0.x release is experimental.