Public source-finder outputs
This reference defines the four-file bundle returned by
hebog.find_sources(). It is for astronomers evaluating measurements and for
developers consuming Hebog from another workflow. For the processing decisions
that produce these files, see How Hebog works.
Hebog is experimental and scientifically unqualified. A successful call means that Hebog completed and validated its output contract; it does not mean that the selected thresholds or algorithms are qualified for a survey.
Product set at a glance
| File | Role and format | Scientific question it answers |
|---|---|---|
catalogue.fits |
source-catalogue, FITS schema 4 |
What islands, associated sources, and admitted Gaussian components were measured? |
rms.fits |
rms, FITS image schema 1 |
What local RMS did thresholding and reported local-noise fields use? |
source-mask.fits |
source-filtering-mask, FITS image schema 1 |
Which input-aligned pixels belong to retained published detections? |
diagnostics.json |
diagnostics, JSON schema 13 |
What was omitted, deferred, selected, or unavailable, and exactly which science produced the bundle? |
flat-noise-rms.fits |
rms, FITS image schema 1; only when the request names a flat-noise image |
What local RMS does the sector's flat-noise image have? |
Use the first four together. In particular, catalogue.fits contains only published
measurements, while diagnostics.json is the census of measured and
unmeasurable component/source identities.
The returned result record
SourceFinderResult schema 3 is an immutable, scheduler-safe summary. It
contains run_id, source_count, gaussian_component_count, island_count,
wall_seconds, four MaterializedProduct records named catalogue, rms,
mask, and diagnostics, and flat_noise_rms, the flat-noise image's RMS
product or null when the request named no flat-noise image.
A sector's flat-noise image
SourceFinderRequest schema 2 accepts flat_noise_image_path: the
primary-beam-uncorrected image of the same sector, as Rapthor's imaging
writes it beside the primary-beam-corrected one. It must share the true-sky
image's shape and celestial WCS, or the run stops before any product is
written, and supplied_metadata applies to both. The run analyses the
true-sky image as usual, then estimates the flat-noise image's background
and RMS on the same executor and with the same configuration, and publishes
its RMS as flat-noise-rms.fits. Nothing else is measured on the flat-noise
image, as PyBDSF's second pass in Rapthor publishes only its RMS. The
catalogue, RMS and mask do not depend on whether a flat-noise image is
given.
Every MaterializedProduct contains:
| Field | Meaning |
|---|---|
product_role |
Semantic role; do not infer this from the filename. |
path |
Closed local file path, rebased to the published bundle. |
media_type |
application/fits, image/fits, or application/json. |
byte_count |
Exact file size used for restart validation. |
content_sha256 |
Lowercase SHA-256 of the complete file bytes. |
scientific_status |
valid, or unavailable for the RMS product only. |
content_schema_version |
Version of the referenced file's content. |
schema_version |
Version 1 of the materialized-product record itself. |
The convenience properties catalogue_path, rms_path, mask_path, and
diagnostics_path return paths only. Pass the full product record to a Hebog
reader when possible: it lets the reader verify role, schema, byte count, and
SHA-256 before parsing.
Catalogue FITS
Catalogue populations and relationships
The primary HDU carries no data array. It identifies the product with:
| Header | Meaning |
|---|---|
HBGROLE=CATALOGUE |
Hebog product role. |
HBGSCHE=4 |
FITS serialization schema. |
CATID |
Stable catalogue identifier derived from the run ID. |
HBGFRAME=icrs |
Coordinate frame. |
HBGEPCH=J2000.0 |
Position epoch. |
RESTFRQ |
Common reference frequency in Hz. |
Every HDU also carries standard FITS CHECKSUM and DATASUM cards. These
detect FITS-level corruption; the result record's SHA-256 identifies the
complete file bytes.
The file then contains exactly three binary-table extensions:
ISLANDS: eight-connected components of the published mask;SOURCES: measured image-domain associations; andGAUSSIAN_COMPONENTS: admitted fitted Gaussian models whose parent source is also published.
Identifiers are stable domain strings and rows are in canonical identifier order. A source links to one primary island and zero or more additional islands. A Gaussian component links to one source and a subset of that source's islands: those its own retained support reaches or, for a component deblended onto a brighter one's rim outside the mask, those of the detection it was split from. These relations permit all of the following:
- one island containing several independent sources;
- one source containing several Gaussian components;
- one source spanning disconnected islands; and
- a source with no Gaussian component because its source-aperture measurement succeeded while a component fit did not.
An island is segmentation, a Gaussian component is a model, and a source is an association hypothesis. None is an assertion of astrophysical truth.
ISLANDS fields
| Column | Unit | Interpretation |
|---|---|---|
ISLAND_ID |
— | Stable identifier for one connected published-mask footprint. |
PIXEL_COUNT |
pixel | Number of 1 pixels in that footprint. |
INTEGRATED_FLUX |
Jy | Signed sum of original background-subtracted footprint pixels divided by the Gaussian restoring-beam area in pixels. It is threshold-support photometry, not a fitted-model integral. |
INTEGRATED_FLUX_ERROR |
Jy | The public finder leaves this unavailable. |
LOCAL_RMS |
Jy/beam | Median positive finite RMS over the footprint. |
MEAN_BRIGHTNESS |
Jy/beam | Mean original background-subtracted brightness over the footprint. |
Island integrated flux and mean brightness may be non-positive. The island can still be a valid segmentation record even when no positive source measurement is publishable.
Fields shared by SOURCES and GAUSSIAN_COMPONENTS
| Column or group | Unit | Interpretation |
|---|---|---|
ISLAND_ID |
— | Primary connected mask island. |
ADDITIONAL_ISLAND_IDS |
— | Comma-separated canonical additional island IDs; blank means none. |
RIGHT_ASCENSION, DECLINATION |
deg | ICRS position, converted from FK5 J2000 when the input uses that frame. RA is in [0, 360) and declination in [-90, 90]. |
RIGHT_ASCENSION_ERROR, DECLINATION_ERROR |
deg | Optional one-sigma uncertainties; FITS NaN means unavailable. |
PEAK_FLUX |
Jy/beam | Peak brightness for the row's estimator. |
PEAK_FLUX_ERROR |
Jy/beam | Optional one-sigma uncertainty. |
INTEGRATED_FLUX |
Jy | Integrated flux under the row-specific estimator described below. |
INTEGRATED_FLUX_ERROR |
Jy | Optional one-sigma uncertainty. |
LOCAL_RMS |
Jy/beam | Median local RMS over the row's owned support. |
SPECTRAL_KIND |
— | Currently reference-frequency-only; no spectral fit is implied. |
REFERENCE_FREQUENCY |
Hz | Frequency at which the flux is reported; it must equal primary RESTFRQ. |
SPECTRAL_COEFFICIENTS |
— | Fixed-width float64 vector. It is empty for the current reference-frequency-only model; trailing NaN is serialization padding. |
FITTED_MAJOR, FITTED_MINOR, FITTED_POSITION_ANGLE |
deg | Optional fitted FWHM ellipse and astronomical position angle in [0, 180). |
FITTED_MAJOR_ERROR, FITTED_MINOR_ERROR, FITTED_POSITION_ANGLE_ERROR |
deg | Optional one-sigma fitted-shape errors. |
DECONVOLVED_MAJOR, DECONVOLVED_MINOR, DECONVOLVED_POSITION_ANGLE |
deg | Optional restoring-beam-deconvolved FWHM ellipse. |
DECONVOLVED_MAJOR_ERROR, DECONVOLVED_MINOR_ERROR, DECONVOLVED_POSITION_ANGLE_ERROR |
deg | Optional one-sigma deconvolved-shape errors. |
QUALITY_FLAGS |
— | Comma-separated canonical interpretation/provenance flags; blank means none. |
Optional floating-point fields use FITS NaN only at the FITS boundary. The
validated Python records decode them to None; a missing uncertainty is not
zero. A major-axis-only deconvolution stores the positive major axis, leaves
the minor axis and angle NaN, and includes major-axis-only. It is not a
complete ellipse.
Source-only fields and measurement meaning
SOURCE_ID is the source identity. Source measurement semantics depend on the
explicit profile:
- In
continuum,INTEGRATED_FLUXis the sum of the source's fitted Gaussian components' integrated fluxes, which is how PyBDSF defines a source's total flux.INTEGRATED_FLUX_ERRORis the quadrature sum of the component uncertainties, published only when every component supplies one. A source with no admitted fit keeps its signed source-owned aperture measurement and carriesaperture-flux-without-fitted-component.ASSOCIATION_APERTURE_FLUXreports the aperture quantity, whether or not the source flux is taken from it. Where the aperture sums to zero or below, as it can where neighbouring sources' wings raise the background, the column is empty and the row carriesassociation-aperture-nonpositive; only a source with an admitted fit has a flux to publish then. A source of one component with an admitted fit is published as that Gaussian, as PyBDSF publishes a single-Gaussian source: its position, position errors,PEAK_FLUXand its error, and fitted and deconvolved shape are the component's, with the flags that qualify them, includingoriginal-pixel-gaussian-model. The errors then describe the position published. Any other source, of several components or of one without a fit, keeps its centroid, no position errors and no claimed shape, and itsPEAK_FLUXis the brightest background-subtracted pixel it owns, not a fitted peak. - In
compact, every published source represents exactly one fitted component.INTEGRATED_FLUXis its Gaussian-model integral andASSOCIATION_APERTURE_FLUXis unavailable.
What the two source fluxes measure, and where they part
The aperture measures emission observed inside the source's own footprint. The summed fit integrates each fitted Gaussian over the whole plane. They agree on isolated compact sources and part elsewhere; the divergence is measured, documented and deliberately not corrected, because PyBDSF's definition behaves the same way and Rapthor's photometry floor was set against it.
| Injected morphology | Aperture against truth | Summed fit against truth |
|---|---|---|
| Isolated compact Gaussian | −0.0% | −0.0% |
| Open arc, several components | −0.0% | +7.4% |
| Gaussian clipped by the image edge | −0.0% | +72.6% |
| Compact core with a diffuse halo | −4.8% | −61.7% |
A fit integrates sky the image does not cover, so an edge-clipped source
reads high. A diffuse halo that no component fits contributes nothing to the
sum, so extended emission reads low even though the aperture retains it.
Read ASSOCIATION_APERTURE_FLUX when observed flux inside the footprint is
the quantity you want, and INTEGRATED_FLUX when PyBDSF-comparable
photometry is.
Free Gaussian fitting also carries a signal-to-noise-dependent positive
bias, inherent to the estimator rather than to Hebog: a median excess over
injected truth of +12.8% at SNR 10, +2.0% at SNR 20 and +0.3% at SNR 50 on
independent noise realizations, within the plan's limits of +14%, +3.5% and
+1%, with pinned PyBDSF master measured on the same population. It is
documented rather than corrected; the 3σ-clipped mean ratio to truth is
within 2% at SNR ≥ 20.
Source apertures are formed from source ownership and adjacent-scale persistent support, then expanded by a bounded aperture. Competing source apertures are non-overlapping: each observable pixel contributes to at most one source. Measurement-only wings can contribute flux but do not move the position footprint. The position of a source of several components is selected from signed original pixels or the denoised residual according to the recorded position rule. It may lie between peaks or in the centre of a shell; it is not a host-galaxy identification. A source of one fitted component publishes the fit's position instead, and its source disposition still records the centroid it would have had.
Continuum source rows of several components do not claim a Gaussian shape,
because an irregular signed aperture describes them. FITTED_* and
DECONVOLVED_* are therefore unavailable on them even when the source flux
and position are valid. A non-positive or otherwise unavailable signed
measurement is not replaced by positive-only photometry. Where only the aperture sum is non-positive, a
fitted source keeps its summed fit, as above, and a source without an
admitted fit has no flux and no row. A source whose position cannot be
measured has no row, fitted or not. Diagnostics retain the identity and
reason of every source without a row. Compact-profile source rows carry the
fitted and deconvolved shape of their one component when available.
Gaussian-component fields and measurement meaning
GAUSSIAN_COMPONENT_ID identifies one admitted fit and SOURCE_ID identifies
its parent source. INTEGRATED_FLUX is the infinite-plane integral of the
selected Gaussian model, including the model's off-image tail. PEAK_FLUX is
its fitted peak amplitude. Every Gaussian-component row has a complete fitted
ellipse.
Fits use original background-subtracted pixels and may be solved jointly where
component contexts interact; a group too large for one joint fit is fitted
island by island. An admitted fit is free elliptical or beam constrained; the
exact model and fallback evidence live in the component's diagnostic
disposition. Failure to admit a fit removes the Gaussian row. It does not
remove the detection footprint or an independently measurable continuum
source. An admitted fit is also omitted if its parent source measurement
cannot be published; the component disposition still records that the fit
succeeded and sets catalogue_row_published to false.
Interpreting quality flags
Flags are canonical, extensible labels rather than a fixed bit mask. Consumers must preserve unknown flags. Common current categories include:
| Examples | Meaning |
|---|---|
original-pixel-gaussian-model, joint-gaussian-fit |
Estimator or fit-context provenance. |
resolved, unresolved, major-axis-only, marginal-deconvolution |
Restoring-beam deconvolution state. |
extension-not-significant, major-axis-not-significant, minor-axis-not-significant |
Why geometric extension was not fully admitted. |
uncertainty-unavailable, shape-uncertainty-unavailable, position-flux-uncertainty-unavailable, deconvolution-uncertainty-unavailable |
Which uncertainty calculation was unavailable. |
fit-at-bound, beam-constrained-fit, free-model-not-significantly-extended |
Selected-model and fallback evidence; consult diagnostics for the structured decision. |
reconstructed-catalogue-source, shape-unavailable, resolution-unavailable, ambiguous-multiscale-parent |
Associated-source construction and interpretation. |
aperture-flux-without-fitted-component |
The continuum source has no admitted fit, so INTEGRATED_FLUX is its aperture rather than a summed fit. |
association-aperture-nonpositive |
The continuum source's aperture summed to zero or below, so ASSOCIATION_APERTURE_FLUX is empty; INTEGRATED_FLUX is its summed fit. |
member-... |
A source-level propagation of a member component's flag; it does not change the source estimator. |
Use structured diagnostics, not string parsing, for workflow decisions about availability, estimator, model selection, or association.
RMS FITS image
rms.fits is a two-dimensional float64 image aligned with the input celestial
WCS; flat-noise-rms.fits has the same form, from the flat-noise image. It copies the restoring beam and reference frequency metadata and uses:
| Header | Value or meaning |
|---|---|
BUNIT |
Jy/beam |
HBGROLE |
RMS |
HBGSCHE |
1 |
HBGSTAT |
VALID or UNAVAILABLE |
For a valid product, finite pixels are non-negative and invalid/unmeasurable
locations may be NaN. Every invalid input pixel is NaN: NaN, BLANK, and
every pixel of a block of one repeated value (see the
input header contract). A valid
status means that at least one finite estimate exists; it does not claim that
every image position has independently resolved noise information. The
continuum estimator can interpolate missing grid cells and extend fine-grid
edge values under its documented policy.
Where it measures local noise (the continuum profile on images at least
150 pixels on a side), a fine cell whose window overlaps a protected source
takes its nearest clean window's RMS, but, while a clean window lies within
half a coarse window of it, never less than 0.8 of the coarse RMS at its
centre or of the largest such clean window if that is lower. So a quieter
region's noise is not carried across a sharp step in the noise beside it,
and extended emission that an unprotected coarse window keeps is not taken
for noise: a cell beyond every clean window's reach, as inside a source
wider than a coarse window, keeps its nearest clean window's RMS, and a cell
a clean window measured keeps its own.
Around bright sources the fine grid only raises the coarse RMS, and carries a clean window's RMS at most one fine window (35 pixels): a fine cell farther than that from every clean window keeps the coarse RMS. Beside a sharp step in the noise of a crowded field the RMS is therefore the coarse estimate, which mixes the two sides over about half a coarse window. Towards the image edge the fine estimate is held at its edge cells, and the coarse estimate never falls below the edge cell, so no extended edge value is lower than the cell beside it.
A noise window measures no noise when its clipped spread is no greater than its noise floor: the largest absolute valid value in the window times single precision's machine epsilon, 2⁻²³ or about 1.2×10⁻⁷, which is single precision's resolution at that value. Samples that all hold one value, such as zero rows between NaN rows, fall under it, and so do differences finer than single precision resolves beside the window's brightest pixel, such as the tails of a noise-free source in a window that holds the source. Such a window is dropped, as a window over a protected source is, and takes the estimate of its nearest clean window under the floor and reach above, so a valid product holds no RMS of zero or of rounding. Real noise falls under the floor only in a window whose brightest pixel is more than eight million times the noise.
A noise-free image whose windows all fall under the floor publishes an unavailable RMS, but a window that holds only a source's far tails, whose spread is close to their own values, is measured, so a noise-free image can still publish a positive but tiny RMS.
An unavailable RMS product is entirely NaN. In that case the catalogue is empty and the source mask is zero because sigma thresholding was not defined. Do not treat NaN as zero noise, invent a floor, or interpret the empty catalogue as a non-detection statement.
The background estimate is not published in the current public bundle.
rms.fits must not be interpreted as a background, residual, variance, weight,
or primary-beam response image.
Source-filtering mask FITS image
source-mask.fits is a two-dimensional uint8 image aligned with the input WCS:
1means the pixel belongs to retained publication support;0means it does not; andBUNIT=1,HBGROLE=MASK,HBGSCHE=1, andHBGSTAT=VALIDdescribe the product.
The mask is a detection footprint. It is not a clean mask for deconvolution,
a component-label map, a source-owner map, a Gaussian model, or the complete
photometric aperture. Connected 1 pixels define catalogue islands, but
source association is independent of mask connectivity.
Invalid input pixels cannot be members. Persistent multiscale support is admitted only under the governed boundary and ownership rules, and joins a detection's footprint only through that detection's own pixels; it does not mean every low-surface-brightness pixel near a source is included. Those rules trim a detection's footprint but never remove an accepted detection: one they would remove entirely keeps its whole thresholded footprint.
Diagnostics JSON
diagnostics.json is canonical UTF-8 JSON with sorted keys and one final
newline. Schema 13 rejects unknown fields and contains:
| Field | Meaning |
|---|---|
run_id |
Caller-provided run identity. |
profile |
continuum or compact. |
profile_limitations |
Empty for continuum; extended-emission-incomplete for compact. |
configuration_qualification |
development-unqualified for exact 5/3-sigma, seven-pixel settings with no maximum; otherwise custom-unqualified. Neither means scientifically qualified. |
source_count, gaussian_component_count, island_count |
Counts that must agree with the FITS tables. |
deblended_parent_count |
Retained connected parents split by bounded deblending. |
deferred_deblend_parent_count |
Parents preserved because they exceeded the bounded deblend envelope. |
wide_object_counts |
How many objects each round decided from the cores that hold them rather than from one window. publication_owners, support_components, islands and segments count objects whose window exceeded a task's read budget; the driver then holds those objects' own pixels or seeds, the one term that grows with an object rather than a tile. deferred_fit_parents answers to a different bound: it counts parents the fit stage deferred because their direct window exceeded the science profile's compact admission bound (maximum_compact_bounds_pixels), whatever the read budget. Such a parent's component records are gathered from the cores, no Gaussian model is fitted to it, and its components carry deferred measurement dispositions. All zero for any image within the public envelope observed so far. |
bright_regions |
How far the true-sky background estimate refined the noise around bright sources. candidate_count counts the bright candidates it found, at or above its bright-candidate threshold, 75σ for any island threshold below that. Candidates close enough to chain merge into one region, which can span the image; its fine cells are estimated in blocks that each read a bounded context, and largest_read_pixels is the most pixels one of those tasks read, zero when none read any. Both figures are evidence for the memory a bright, crowded field needs, not science. |
flat_noise_bright_regions |
The same record for the flat-noise image's estimate; present exactly when the request named one. |
measurement_dispositions |
Complete structured census described below. |
rms_scientific_status |
valid or unavailable, matching the RMS product. |
provenance |
Exact input, configuration, science-profile, and implementation identities, and any caller-supplied image metadata. |
schema_version |
13. |
Provenance
| Field | Meaning |
|---|---|
input_sha256 |
Bytes of the input FITS file. |
flat_noise_input_sha256 |
Bytes of the flat-noise image, or null when the request named none. |
configuration_sha256 |
Canonical complete SourceFinderConfig, including thresholds, size limits, and profile. |
scientific_profile_sha256 |
Exact installed science-configuration resource. |
scientific_composition |
Opaque implementation label. Preserve it for provenance; users do not need to interpret it. |
scientific_composition_sha256 |
Exact identity of the implementation: every Hebog module the finder imports, except the package initializer, tile planning, the tasks' memory declarations and the source-count estimate a caller runs before an analysis, and every packaged resource file. |
supplied_image_metadata |
null, or the SuppliedImageMetadata values the request supplied for keywords the input header omits: reference_frequency_hz, beam_major_fwhm_degrees, beam_minor_fwhm_degrees, beam_position_angle_degrees and brightness_unit, each null when not supplied. The input SHA-256 alone does not identify a run that used supplied metadata. |
schema_version |
4 for the nested provenance record. |
Two runs should be treated as the same scientific computation only after the relevant identities, software/environment context, and product bytes have been compared. The run ID alone is not a science identity.
Measurement dispositions
There is one disposition for every established component and source identity, including objects with no published catalogue row.
| Field | Meaning |
|---|---|
object_kind, object_id |
component or source, and its stable identity. |
status |
measured, unavailable, or deferred. |
estimator |
original-pixel-gaussian-model for a measured component. For a measured continuum source, summed-fitted-component-flux, or source-owned-signed-aperture when it has no admitted fit; a compact source's disposition is its one component's, with the component's identity and estimator. Otherwise null. |
reason |
Null for a measurement; explicit cause for unavailable/deferred work. |
member_component_ids |
Complete canonical membership for a source; empty for a component. Source memberships partition the component population. |
catalogue_row_published |
Whether this exact identity appears in its corresponding FITS table. |
fit_covariance_available |
Whether an admitted component fit has formal covariance; null when not applicable. |
fit_diagnostics |
Optimizer, model, bounds, conditioning, visibility, and noise-estimator evidence. |
position_diagnostics |
Source centroid alternatives, selection rule, counts, signed weights/flux, and background estimate. |
association_diagnostics |
Competing hierarchy, compact-model, and extended-morphology group IDs plus the selected decision. |
association_evidence |
Evidence stored once for each admitted multi-component merge. |
The number of source dispositions whose catalogue_row_published is true
equals source_count; the equivalent component count equals
gaussian_component_count. This makes it possible to distinguish "not
detected", "detected but not measurable", "measurement deferred", and
"measured and published".
fit_diagnostics records convergence, function evaluations, chi-squared and
degrees of freedom, bound contact, selected/rejected model identities,
condition number, covariance parameterization, visible-model fraction,
retained bounds/pixel count, the point estimator, and any estimator or model
fallback reason. These fields
describe estimator behaviour; they are not independent source-quality scores.
position_diagnostics records signed-original, denoised, and selected pixel
positions in (x, y) order, the selection and unavailable reasons, position
and aperture pixel counts, signed weight, signed aperture flux in Jy, and the
estimated aperture background mean. It does not estimate astrophysical host
position or background truth.
association_evidence names the reason (directional-fwhm-overlap,
resolved-loop, resolved-open-arc, or persistent-residual), contributing
scale IDs, all member components, and any overridden compact protection. It is
evidence for an image-domain grouping decision, not confirmation that the
members are one physical object.
Fit and association-decision diagnostics apply to component dispositions. Position diagnostics and association evidence apply to source dispositions. Fields that are not meaningful for that object or outcome are null or empty; consumers should not substitute zero or an inferred default.
Reading and validating the bundle
Use the public product records rather than bare paths when consuming an in-process result:
from hebog.data_models import ImageBounds
from hebog.io import (
FitsProductImageSource,
read_catalogue_fits_product,
read_diagnostics_product,
)
catalogue = read_catalogue_fits_product(result.catalogue)
diagnostics = read_diagnostics_product(result.diagnostics)
rms_source = FitsProductImageSource(result.rms)
mask_source = FitsProductImageSource(result.mask)
height, width = rms_source.metadata().shape_yx
full_image = ImageBounds(0, height, 0, width)
rms = rms_source.read_window(full_image).values
mask = mask_source.read_window(full_image).values.astype(bool)
The readers fail on a mismatched role, unsupported schema, changed byte count
or SHA-256, malformed FITS structure, noncanonical diagnostics JSON, invalid
units, negative RMS, or non-binary mask. Bounded ImageBounds windows avoid
forcing an integrating pipeline to load a complete image product.
Consumers outside Python can read the exact FITS structure above, but should
still preserve unknown quality flags, check HBGROLE, HBGSCHE, and
HBGSTAT, and verify the result record's byte identity before trusting a
persisted product.
Failure handling
Configuration construction raises ValueError for a threshold that is not a
number, invalid threshold ordering, pixel limits, or profile names. A NumPy
scalar is accepted wherever a number is. Once find_sources() is called, its
public exceptions let a pipeline handle failures without parsing message
text:
| Exception | Meaning |
|---|---|
SourceFinderOutputExistsError |
The caller-owned output path already exists and does not hold this request's finished products (running a finished request again); Hebog will not overwrite it. Publication claims the destination atomically, so a path another writer creates while the analysis runs is reported here rather than replaced. Products then appear in one rename; treat a successful return, not the directory's existence, as the completion boundary. |
InvalidSourceFinderInputError |
The FITS file cannot be read as a supported image: it is missing, truncated or not FITS, it is not one plane, or a header card the finder reads is missing, malformed or ambiguous. The message names the keyword or layout. A unit, beam or frequency that the header lacks and the request does not supply is reported here. |
UnsupportedSourceFinderConfigurationError |
The image is readable but outside what the requested science runs on: a unit other than Jy/beam, a celestial frame other than ICRS or FK5 J2000, or a restoring beam wider than 10 pixels. |
SourceFinderImageTooLargeError |
A spatial dimension exceeds 22,500 pixels, or the shorter side is under 600 pixels and the image holds more than 1,000,000. |
SourceFinderError |
Base class for other failures at the public boundary. |
The input header contract states each rule. All three input errors are raised before the analysis starts, and the input file is closed when the error reaches the caller. Input, configuration, and existing-output failures do not publish the requested bundle. Analysis or product-validation failures use the same all-or-nothing directory boundary, so the caller can retry after addressing the cause.
What a failed or killed run leaves
A run builds its work planes and its product bundle in a hidden staging
directory beside the output, .<output name>.<random>, and removes it before
find_sources() returns or raises. A failure inside an executor task reaches
the caller as that task's own exception, and only once no task the run
submitted is still running, so no late write can recreate the staging
directory: an executor submits nothing more after a failure, cancels what
has not started and waits for what has. Dask cannot stop a task a worker has
started, and a Dask future does not say whether its task has started, so
DaskExecutor waits for every task it has in flight, at most
capacity.maximum_tasks_in_flight. A task that never finishes holds the
call open, as it would in a successful run.
A caller's deadline ends a run the same way when it raises in the thread
that called find_sources(), as a Prefect task timeout does off the main
thread: the run raises the caller's exception once no task it started is
still running, starts none afterwards, and removes its staging directory.
ThreadExecutor holds each call's tasks behind a guard, so even a task the
pool queued as the exception arrived never runs. With DaskExecutor, an
exception that arrives while a task is being handed to the scheduler can
leave that one task running unobserved; one that arrives while the run
waits, where it spends its time, cannot.
A process that is killed cannot remove its staging directory, so the
directory records its owner. owner.json names the output, the run
identifier, the host, the process identifier and the start time, and the
owner holds an exclusive lock on owner.lock while it runs. The operating
system releases that lock when the process ends, however it ends. Before it
starts, the next run to the same output inspects each staging directory
beside it:
| Staging directory found | What the next run does |
|---|---|
| Recorded on this host, and its lock is free | Removes it: the owner has stopped. |
| Its owner still holds the lock | Leaves it in place: the owner is running. |
| Recorded on another host, or by an owner whose filesystem could not lock | Leaves it in place: a lock seen from here proves nothing about that owner. |
| No owner record this version of Hebog reads | Leaves it in place. |
Each is reported with a SourceFinderStagingWarning naming the directory,
and its owner when one is recorded, and the run continues in a staging
directory of its own. Only a run to the same output looks: a retry under a
new output name leaves an earlier attempt's directory unreported. A
directory left in place stays until its owner finishes or someone removes
it; remove one only once the run it names has stopped. A killed run
publishes nothing, so the same request can run again.
The lock proves only that the process that called find_sources() has
stopped. Tasks it had already submitted to a Dask cluster that outlives it
still run to completion and may write into the directory after the next run
has removed it. What they recreate holds no owner record, so it is reported
and left in place. Before running the request again after killing the
process, let the cluster finish those tasks or restart its workers.
Running a finished request again
A bundle is published in one rename, so one in place is complete. When a
request runs again after its run finished, as when a workflow retries a
task that failed after the finder returned, find_sources() returns the
products already published and analyses nothing. The request must be the
same one: the same run ID, and the same provenance, which binds the input
bytes (and the flat-noise image's), the configuration and the science
composition. Each product it should hold is validated as its writer
validated it, and the returned records equal the first run's apart from
wall_seconds. A bundle of another request, or one missing or damaging a
product, raises SourceFinderOutputExistsError and is kept. A run that
failed part-way published nothing, so its retry analyses the image again.
Evaluation checklist for astronomers
Before using a bundle as scientific evidence:
- Confirm the input identity, implementation identity, profile, configuration
identity, and
configuration_qualificationin diagnostics. - Inspect the RMS image and its status; do not evaluate completeness where sigma thresholding was unavailable or the estimator resolution is inappropriate for the noise structure.
- Overlay
source-mask.fitson the original and background-subtracted image, especially around edges, invalid pixels, close blends, and diffuse emission. - Compare like populations: Hebog
GAUSSIAN_COMPONENTSagainst component catalogues, and HebogSOURCESagainst source-level catalogues. Do not use source count alone as a component-recovery metric. - Examine dispositions for missing Gaussian rows, unavailable signed source measurements, bounded-work deferrals, model fallback, and covariance availability.
- Treat low-S/N crossings as completeness/reliability changes and stratify astrometry, peak flux, integrated flux, and shape residuals by S/N, morphology, edge distance, crowding, and invalid-pixel context.
- Remember that current successful products are development outputs, not a general qualification claim or a validated replacement for every PyBDSF mode.
Integration checklist for developers
- Construct a new caller-owned output directory for each attempt; Hebog never overwrites an existing path.
- Keep
SourceFinderRequestandSourceFinderResultas small path-and-metadata records. Do not pass open FITS handles, mutable image arrays, or scheduler clients through workflow state. - Supply
SerialExecutor()for the deterministic reference orDaskExecutor(existing_client)when the workflow owns a Dask client. Hebog does not create or close the cluster. - Branch on public exception types, product roles, schema versions, and scientific status rather than error-message text or filenames.
- Wait for
find_sources()to return before exposing the bundle. Its final directory rename is the completeness boundary. - Persist the complete result record with workflow provenance. A copied file path without its role, schema, byte count, and SHA-256 loses restart safety.
- Pin an exact Hebog
0.xversion and review schema/release notes before upgrading; the project does not promise backward compatibility between experimental releases.