Averaging¶
Routines for averaging visibility data.
Time and Channel Averaging¶
The routines in this section average row-based samples by:
- Averaging samples of consecutive time values into bins defined
by an period of
time_bin_secs
seconds. - Averaging channel data into equally sized bins of
chan_bin_size
.
In order to achieve this, a baseline x time ordering is established over the input data where baseline corresponds to the unique (ANTENNA1, ANTENNA2) pairs and time corresponds to the unique, monotonically increasing TIME values associated with the rows of a Measurement Set.
Baseline | T0 | T1 | T2 | T3 | T4 |
---|---|---|---|---|---|
(0, 0) | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 |
(0, 1) | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 |
(0, 2) | 0.1 | 0.2 | X | 0.4 | 0.5 |
(1, 1) | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 |
(1, 2) | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 |
(2, 2) | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 |
It is possible for times or baselines to be missing. In the above example, T2 is missing for baseline (0, 2).
Warning
The above requires unique lexicographical combinations of (TIME, ANTENNA1, ANTENNA2). This can usually be achieved by suitably partitioning input data on indexing rows, DATA_DESC_ID and SCAN_NUMBER in particular.
For each baseline, adjacent time’s are assigned to a bin
if \(h_c - h_e/2 - (l_c - l_e/2) <\) time_bin_secs
, where
\(h_c\) and \(l_c\) are the upper and lower time and
\(h_e\) and \(l_e\) are the upper and lower intervals,
taken from the INTERVAL column.
Note that no distinction is made between flagged and unflagged data
when establishing the endpoints in the bin.
The reason for this is that the Measurement Set v2.0 Specification specifies that TIME and INTERVAL columns are defined as containing the nominal time and period at which the visibility was sampled. This means that their values includie valid, flagged and missing data. Thus, averaging a regular high-resolution baseline x htime grid should produce a regular low-resolution baseline x ltime grid (htime > ltime) in the presence of bad data
By contrast, other columns such as TIME_CENTROID and EXPOSURE contain the effective time and period as they exclude missing and bad data. Their increased accuracy, and therefore variability means that they are unsuitable for establishing a grid over the data.
To summarise, the averaged times in each bin establish a map:
- from possibly unordered input rows.
- to a reduced set of output rows ordered by
averaged
(TIME, ANTENNA1, ANTENNA2)
.
Flagged Data Handling¶
Both FLAG_ROW and FLAG columns may be supplied to the averager, but they should be consistent with each other. The averager will throw an exception if this is not the case, rather than making an assumption as to which is correct.
When provided with flags, the averager will output averages for bins that are completely flagged.
Part of the reason for this is that the specifies that the TIME and INTERVAL columns represent the nominal time and interval values. This means that they should represent valid as well as flagged or missing data in their computation.
By contrast, most other columns such as TIME_CENTROID and EXPOSURE, contain the effective values and should only include valid, unflagged data.
To support this:
- TIME and INTERVAL are averaged using both flagged and unflagged samples.
- Other columns, such as TIME_CENTROID are handled as follows:
- If the bin contains some unflagged data, only this data is used to calculate average.
- If the bin is completely flagged, the average of all samples (which are all flagged) will be used.
- In both cases, a completely flagged bin will have it’s flag set.
- To support the two cases, twice the memory of the output array is required to track both averages, but only one array of merged values is returned.
Guarantees¶
- Averaged output data will be lexicographically ordered by
(TIME, ANTENNA1, ANTENNA2)
- TIME and INTERVAL columns always contain the nominal average and sum and therefore contain both and missing or unflagged data.
- Other columns will contain the effective average and will contain only valid data except when all data in the bin is flagged.
- Completely flagged bins will be set as flagged in both the nominal and effective case.
- Certain columns are averaged, while others are summed, or simply assigned to the last value in the bin in the case of antenna indices.
- Visibility data is averaged by multiplying and dividing by WEIGHT_SPECTRUM or WEIGHT or natural weighting, in order of priority.
\[\frac{\sum v_i w_i}{\sum w_i}\]
SIGMA_SPECTRUM is averaged by multiplying and dividing by WEIGHT_SPECTRUM or WEIGHT or natural weighting, in order of priority and availability.
SIGMA is only averaged with WEIGHT or natural weighting.
\[\sqrt{\frac{\sum w_i^2 \sigma_i^2}{(\sum w_i)^2}}\]
The following table summarizes the handling of each column in the main Measurement Set table:
Column | Unflagged/Flagged sample handling | Aggregation Method | Required |
---|---|---|---|
TIME | Nominal | Mean | Yes |
INTERVAL | Nominal | Sum | Yes |
ANTENNA1 | Nominal | Assigned to Last Input | Yes |
ANTENNA2 | Nominal | Assigned to Last Input | Yes |
TIME_CENTROID | Effective | Mean | No |
EXPOSURE | Effective | Sum | No |
FLAG_ROW | Effective | Set if All Inputs Flagged | No |
UVW | Effective | Mean | No |
WEIGHT | Effective | Sum | No |
SIGMA | Effective | Weighted Mean | No |
DATA (vis) | Effective | Weighted Mean | No |
FLAG | Effective | Set if All Inputs Flagged | No |
WEIGHT_SPECTRUM | Effective | Sum | No |
SIGMA_SPECTRUM | Effective | Weighted Mean | No |
The following SPECTRAL_WINDOW sub-table columns are averaged as follows:
Column | Aggregation Method |
---|---|
CHAN_FREQ | Mean |
CHAN_WIDTH | Sum |
EFFECTIVE_BW | Sum |
RESOLUTION | Sum |
Dask Implementation¶
The dask implementation chunks data up by row and channel and averages each chunk independently of values in other chunks. This should be kept in mind if one wishes to maintain a particular ordering in the output dask arrays.
Typically, Measurement Set data is monotonically ordered in time. To maintain this guarantee in output dask arrays, the chunks will need to be separated by distinct time values. Practically speaking this means that the first and second chunk should not both contain value time 0.1, for example.
Numpy¶
time_and_channel (time, interval, antenna1, …) |
Averages in time and channel. |
-
africanus.averaging.
time_and_channel
(time, interval, antenna1, antenna2, time_centroid=None, exposure=None, flag_row=None, uvw=None, weight=None, sigma=None, chan_freq=None, chan_width=None, effective_bw=None, resolution=None, vis=None, flag=None, weight_spectrum=None, sigma_spectrum=None, time_bin_secs=1.0, chan_bin_size=1)[source]¶ Averages in time and channel.
Parameters: time :
numpy.ndarray
Time values of shape
(row,)
.interval :
numpy.ndarray
Interval values of shape
(row,)
.antenna1 :
numpy.ndarray
First antenna indices of shape
(row,)
antenna2 :
numpy.ndarray
Second antenna indices of shape
(row,)
time_centroid :
numpy.ndarray
, optionalTime centroid values of shape
(row,)
exposure :
numpy.ndarray
, optionalExposure values of shape
(row,)
flag_row :
numpy.ndarray
, optionalFlagged rows of shape
(row,)
.uvw :
numpy.ndarray
, optionalUVW coordinates of shape
(row, 3)
.weight :
numpy.ndarray
, optionalWeight values of shape
(row, corr)
.sigma :
numpy.ndarray
, optionalSigma values of shape
(row, corr)
.chan_freq :
numpy.ndarray
, optionalChannel frequencies of shape
(chan,)
.chan_width :
numpy.ndarray
, optionalChannel widths of shape
(chan,)
.effective_bw :
numpy.ndarray
, optionalEffective channel bandwidth of shape
(chan,)
.resolution :
numpy.ndarray
, optionalEffective channel resolution of shape
(chan,)
.vis :
numpy.ndarray
, optionalVisibility data of shape
(row, chan, corr)
.flag :
numpy.ndarray
, optionalFlag data of shape
(row, chan, corr)
.weight_spectrum :
numpy.ndarray
, optionalWeight spectrum of shape
(row, chan, corr)
.sigma_spectrum :
numpy.ndarray
, optionalSigma spectrum of shape
(row, chan, corr)
.time_bin_secs : float, optional
Maximum summed interval in seconds to include within a bin. Defaults to 1.0.
chan_bin_size : int, optional
Number of bins to average together. Defaults to 1.
Returns: namedtuple
A namedtuple whose entries correspond to the input arrays. Output arrays will be
None
if the inputs wereNone
.Notes
The implementation currently requires unique lexicographical combinations of (TIME, ANTENNA1, ANTENNA2). This can usually be achieved by suitably partitioning input data on indexing rows, DATA_DESC_ID and SCAN_NUMBER in particular.
Dask¶
time_and_channel (time, interval, antenna1, …) |
Averages in time and channel. |
-
africanus.averaging.dask.
time_and_channel
(time, interval, antenna1, antenna2, time_centroid=None, exposure=None, flag_row=None, uvw=None, weight=None, sigma=None, chan_freq=None, chan_width=None, effective_bw=None, resolution=None, vis=None, flag=None, weight_spectrum=None, sigma_spectrum=None, time_bin_secs=1.0, chan_bin_size=1)[source]¶ Averages in time and channel.
Parameters: time :
dask.array.Array
Time values of shape
(row,)
.interval :
dask.array.Array
Interval values of shape
(row,)
.antenna1 :
dask.array.Array
First antenna indices of shape
(row,)
antenna2 :
dask.array.Array
Second antenna indices of shape
(row,)
time_centroid :
dask.array.Array
, optionalTime centroid values of shape
(row,)
exposure :
dask.array.Array
, optionalExposure values of shape
(row,)
flag_row :
dask.array.Array
, optionalFlagged rows of shape
(row,)
.uvw :
dask.array.Array
, optionalUVW coordinates of shape
(row, 3)
.weight :
dask.array.Array
, optionalWeight values of shape
(row, corr)
.sigma :
dask.array.Array
, optionalSigma values of shape
(row, corr)
.chan_freq :
dask.array.Array
, optionalChannel frequencies of shape
(chan,)
.chan_width :
dask.array.Array
, optionalChannel widths of shape
(chan,)
.effective_bw :
dask.array.Array
, optionalEffective channel bandwidth of shape
(chan,)
.resolution :
dask.array.Array
, optionalEffective channel resolution of shape
(chan,)
.vis :
dask.array.Array
, optionalVisibility data of shape
(row, chan, corr)
.flag :
dask.array.Array
, optionalFlag data of shape
(row, chan, corr)
.weight_spectrum :
dask.array.Array
, optionalWeight spectrum of shape
(row, chan, corr)
.sigma_spectrum :
dask.array.Array
, optionalSigma spectrum of shape
(row, chan, corr)
.time_bin_secs : float, optional
Maximum summed interval in seconds to include within a bin. Defaults to 1.0.
chan_bin_size : int, optional
Number of bins to average together. Defaults to 1.
Returns: namedtuple
A namedtuple whose entries correspond to the input arrays. Output arrays will be
None
if the inputs wereNone
.Notes
The implementation currently requires unique lexicographical combinations of (TIME, ANTENNA1, ANTENNA2). This can usually be achieved by suitably partitioning input data on indexing rows, DATA_DESC_ID and SCAN_NUMBER in particular.