pyvisor.analysis.reliability package

Submodules

pyvisor.analysis.reliability.agreement module

Agreement statistics for two (or more) annotation passes.

All functions are pure: they take numpy arrays and return floats or frozen result dataclasses, so they are trivial to unit-test against hand-computed values.

class pyvisor.analysis.reliability.agreement.BlandAltman(bias: float, sd_diff: float, loa_lower: float, loa_upper: float, n: int)[source]

Bases: object

Bias and 95 % limits of agreement for a scalar across clips.

bias: float

loa_lower: float

loa_upper: float

n: int

sd_diff: float

class pyvisor.analysis.reliability.agreement.ConfusionCounts(true_positive: int, false_positive: int, false_negative: int, true_negative: int)[source]

Bases: object

Frame-wise 2×2 counts for a single behaviour (B scored against A).

false_negative: int

false_positive: int

property n_frames: int

true_negative: int

true_positive: int

class pyvisor.analysis.reliability.agreement.PrecisionRecall(precision: 'float', recall: 'float', f1: 'float')[source]

Bases: object

f1: float

precision: float

recall: float

pyvisor.analysis.reliability.agreement.bland_altman(method_a: numpy.ndarray, method_b: numpy.ndarray) → BlandAltman[source]

Bias and 95 % limits of agreement between two scalar series.

One value per clip from each method/observer (e.g. Courtship Index from GameThogram vs BORIS).

pyvisor.analysis.reliability.agreement.cohen_kappa_binary(rater_a: numpy.ndarray, rater_b: numpy.ndarray) → float[source]

Cohen’s κ for one behaviour scored present/absent per frame.

Parameters:

rater_a – Boolean/0-1 vector, one entry per frame.
rater_b – Boolean/0-1 vector, aligned to rater_a.

Returns:

κ, or nan when chance agreement is 1 (the behaviour is constant for both raters, leaving κ undefined).

pyvisor.analysis.reliability.agreement.cohen_kappa_multiclass(state_a: numpy.ndarray, state_b: numpy.ndarray) → float[source]

Cohen’s κ for a single mutually-exclusive state per frame.

Use this when each frame carries exactly one behavioural state (a categorical label); use cohen_kappa_binary() per column for multi-label ethograms where behaviours may co-occur.

pyvisor.analysis.reliability.agreement.confusion_counts(reference: numpy.ndarray, test: numpy.ndarray) → ConfusionCounts[source]: Frame-wise 2×2 counts, treating reference as ground truth.

pyvisor.analysis.reliability.agreement.icc_2_1(scores: numpy.ndarray) → float[source]

Shrout & Fleiss ICC(2,1) for an (n targets × k raters) matrix.

Two-way random-effects, single-measure, absolute agreement — the correct ICC for “do independent observers reproduce the same per-clip value (e.g. Courtship Index)”.

Parameters:: scores – Array of shape (n_targets, n_raters); rows are clips, columns are observers.
Returns:: ICC(2,1). Returns nan if total variance is zero.

pyvisor.analysis.reliability.agreement.match_event_onsets(onsets_reference: numpy.ndarray, onsets_test: numpy.ndarray, tolerance_frames: int) → PrecisionRecall[source]

Match behaviour onsets within ±*tolerance_frames* (greedy nearest).

Each reference onset may match at most one test onset and vice versa. Returns event-level precision/recall/F1 — the statistic for “annotations land on the right frame”.

pyvisor.analysis.reliability.agreement.percent_agreement(rater_a: numpy.ndarray, rater_b: numpy.ndarray) → float[source]: Raw proportion of frames on which the two raters agree.

pyvisor.analysis.reliability.agreement.precision_recall_f1(reference: numpy.ndarray, test: numpy.ndarray) → PrecisionRecall[source]

Precision, recall and F1 of test against reference.

A component is nan when its denominator is zero (e.g. recall when the behaviour never occurs in the reference).

pyvisor.analysis.reliability.annotation_io module

Load heterogeneous annotation exports into a common frame raster.

The shared representation is RasterAnnotation — a frames × behaviours boolean DataFrame plus the frame rate. Everything in agreement and measures consumes that representation, so GameThogram and BORIS passes become directly comparable.

class pyvisor.analysis.reliability.annotation_io.RasterAnnotation(table: pandas.DataFrame, fps: float, source: str)[source]

Bases: object

Frames × behaviours boolean table with a frame rate and source tag.

property behaviours: list[str]

column(behaviour: str) → numpy.ndarray[source]

fps: float

matrix() → numpy.ndarray[source]

property n_frames: int

source: str

table: pandas.DataFrame

pyvisor.analysis.reliability.annotation_io.align(a: RasterAnnotation, b: RasterAnnotation) → tuple[RasterAnnotation, RasterAnnotation][source]

Restrict two annotations to common behaviours and frame length.

Raises if the frame rates differ or the behaviour sets do not overlap — both are signs the two passes are not comparable.

pyvisor.analysis.reliability.annotation_io.load_boris_tabular(path: Path, fps: float, n_frames: int, behaviours: list[str] | None = None) → RasterAnnotation[source]

Rasterise a BORIS “Export events → Tabular events” file.

Handles state behaviours (START/STOP pairs) and point events. Expected columns (case-insensitive): a behaviour column (Behavior), a status column (Status / Behavior type with START/STOP/POINT), and a time column (Time in seconds, or Image index / Frame in frames).

Parameters:

path – BORIS tabular export (.tsv or .csv).
fps – Frames per second, to convert event times to frames.
n_frames – Length of the raster (match the GameThogram pass).
behaviours – Restrict/order the output columns; defaults to all behaviours seen in the file, sorted.

pyvisor.analysis.reliability.annotation_io.load_gamethogram(path: Path, fps: float, labels: list[str] | None = None) → RasterAnnotation[source]: Load a GameThogram export, dispatching on file suffix.

pyvisor.analysis.reliability.figures module

pyvisor.analysis.reliability.measures module

Per-clip behavioural measures derived from a frame raster.

class pyvisor.analysis.reliability.measures.BoutStatistics(n_bouts: 'int', total_frames: 'int', mean_duration_s: 'float', sd_duration_s: 'float')[source]

Bases: object

mean_duration_s: float

n_bouts: int

sd_duration_s: float

total_frames: int

pyvisor.analysis.reliability.measures.bout_statistics(behaviour_vector: numpy.ndarray, fps: float) → BoutStatistics[source]: Count and time the contiguous bouts of one behaviour.

pyvisor.analysis.reliability.measures.courtship_index(raster: numpy.ndarray, behaviour_columns: numpy.ndarray) → float[source]

Courtship Index: proportion of frames courting.

Parameters:

raster – (frames × behaviours) 0-1 matrix.
behaviour_columns – Indices of the columns that count as courtship (e.g. orienting, tapping, wing extension, licking, attempted copulation).

pyvisor.analysis.reliability.measures.latency_to_first(behaviour_vector: numpy.ndarray, fps: float) → float[source]

Seconds until the behaviour first becomes active.

Returns nan if the behaviour never occurs.

pyvisor.analysis.reliability.measures.onset_frames(behaviour_vector: numpy.ndarray) → numpy.ndarray[source]: Frame indices at which the behaviour switches from off to on.

pyvisor.analysis.reliability.measures.proportion_active(raster: numpy.ndarray) → float[source]

Fraction of frames in which the behaviour is active.

With a 1-D behaviour vector this is that behaviour’s index; pass a 2-D (frames × behaviours) slice and the row-wise any to obtain a Courtship Index over a set of courtship behaviours.

pyvisor.analysis.reliability.report module

Study-level orchestration of the agreement statistics.

Every statistic is computed unconditionally for every clip; thresholds and verdicts belong in the manuscript, not here.

class pyvisor.analysis.reliability.report.ClipComparison(clip_id: str, pass_a: RasterAnnotation, pass_b: RasterAnnotation, courtship_behaviours: list[str])[source]

Bases: object

One clip scored by two passes (observers, or GameThogram vs BORIS).

clip_id: str

courtship_behaviours: list[str]

pass_a: RasterAnnotation

pass_b: RasterAnnotation

class pyvisor.analysis.reliability.report.StudySummary(per_behaviour: 'pd.DataFrame', courtship_index: 'pd.DataFrame', icc_courtship_index: 'float', bland_altman_courtship_index: 'ag.BlandAltman')[source]

Bases: object

bland_altman_courtship_index: BlandAltman

courtship_index: pandas.DataFrame

icc_courtship_index: float

per_behaviour: pandas.DataFrame

pyvisor.analysis.reliability.report.behaviour_agreement(comparison: ClipComparison) → pandas.DataFrame[source]: Per-behaviour κ, %-agreement and F1 for a single clip.

pyvisor.analysis.reliability.report.courtship_index_pair(comparison: ClipComparison) → dict[str, float][source]: Courtship Index from each pass for one clip.

pyvisor.analysis.reliability.report.summarise_study(comparisons: list[ClipComparison]) → StudySummary[source]: Run the full battery over every clip and pool the scalar readouts.

pyvisor.analysis.reliability.report.write_tables(summary: StudySummary, output_dir: Path) → None[source]: Write the tidy result tables as CSVs (reviewer-checkable numbers).

pyvisor.analysis.reliability.viz_constants module

Module contents

Reliability and method-comparison toolkit for the validation study.

Loads GameThogram and BORIS annotation exports into a common frame raster (annotation_io), derives per-clip behavioural measures (measures), and computes inter-/intra-observer and tool-vs-tool agreement statistics (agreement).

class pyvisor.analysis.reliability.BlandAltman(bias: float, sd_diff: float, loa_lower: float, loa_upper: float, n: int)[source]

Bases: object

Bias and 95 % limits of agreement for a scalar across clips.

bias: float

loa_lower: float

loa_upper: float

n: int

sd_diff: float

class pyvisor.analysis.reliability.BoutStatistics(n_bouts: 'int', total_frames: 'int', mean_duration_s: 'float', sd_duration_s: 'float')[source]

Bases: object

mean_duration_s: float

n_bouts: int

sd_duration_s: float

total_frames: int

class pyvisor.analysis.reliability.ConfusionCounts(true_positive: int, false_positive: int, false_negative: int, true_negative: int)[source]

Bases: object

Frame-wise 2×2 counts for a single behaviour (B scored against A).

false_negative: int

false_positive: int

property n_frames: int

true_negative: int

true_positive: int

class pyvisor.analysis.reliability.PrecisionRecall(precision: 'float', recall: 'float', f1: 'float')[source]

Bases: object

f1: float

precision: float

recall: float

class pyvisor.analysis.reliability.RasterAnnotation(table: pandas.DataFrame, fps: float, source: str)[source]

Bases: object

Frames × behaviours boolean table with a frame rate and source tag.

property behaviours: list[str]

column(behaviour: str) → numpy.ndarray[source]

fps: float

matrix() → numpy.ndarray[source]

property n_frames: int

source: str

table: pandas.DataFrame

pyvisor.analysis.reliability.align(a: RasterAnnotation, b: RasterAnnotation) → tuple[RasterAnnotation, RasterAnnotation][source]

Restrict two annotations to common behaviours and frame length.

Raises if the frame rates differ or the behaviour sets do not overlap — both are signs the two passes are not comparable.

pyvisor.analysis.reliability.bland_altman(method_a: numpy.ndarray, method_b: numpy.ndarray) → BlandAltman[source]

Bias and 95 % limits of agreement between two scalar series.

One value per clip from each method/observer (e.g. Courtship Index from GameThogram vs BORIS).

pyvisor.analysis.reliability.bout_statistics(behaviour_vector: numpy.ndarray, fps: float) → BoutStatistics[source]: Count and time the contiguous bouts of one behaviour.

pyvisor.analysis.reliability.cohen_kappa_binary(rater_a: numpy.ndarray, rater_b: numpy.ndarray) → float[source]

Cohen’s κ for one behaviour scored present/absent per frame.

Parameters:

rater_a – Boolean/0-1 vector, one entry per frame.
rater_b – Boolean/0-1 vector, aligned to rater_a.

Returns:

κ, or nan when chance agreement is 1 (the behaviour is constant for both raters, leaving κ undefined).

pyvisor.analysis.reliability.cohen_kappa_multiclass(state_a: numpy.ndarray, state_b: numpy.ndarray) → float[source]

Cohen’s κ for a single mutually-exclusive state per frame.

Use this when each frame carries exactly one behavioural state (a categorical label); use cohen_kappa_binary() per column for multi-label ethograms where behaviours may co-occur.

pyvisor.analysis.reliability.confusion_counts(reference: numpy.ndarray, test: numpy.ndarray) → ConfusionCounts[source]: Frame-wise 2×2 counts, treating reference as ground truth.

pyvisor.analysis.reliability.courtship_index(raster: numpy.ndarray, behaviour_columns: numpy.ndarray) → float[source]

Courtship Index: proportion of frames courting.

Parameters:

raster – (frames × behaviours) 0-1 matrix.
behaviour_columns – Indices of the columns that count as courtship (e.g. orienting, tapping, wing extension, licking, attempted copulation).

pyvisor.analysis.reliability.icc_2_1(scores: numpy.ndarray) → float[source]

Shrout & Fleiss ICC(2,1) for an (n targets × k raters) matrix.

Two-way random-effects, single-measure, absolute agreement — the correct ICC for “do independent observers reproduce the same per-clip value (e.g. Courtship Index)”.

Parameters:: scores – Array of shape (n_targets, n_raters); rows are clips, columns are observers.
Returns:: ICC(2,1). Returns nan if total variance is zero.

pyvisor.analysis.reliability.latency_to_first(behaviour_vector: numpy.ndarray, fps: float) → float[source]

Seconds until the behaviour first becomes active.

Returns nan if the behaviour never occurs.

pyvisor.analysis.reliability.load_boris_tabular(path: Path, fps: float, n_frames: int, behaviours: list[str] | None = None) → RasterAnnotation[source]

Rasterise a BORIS “Export events → Tabular events” file.

Handles state behaviours (START/STOP pairs) and point events. Expected columns (case-insensitive): a behaviour column (Behavior), a status column (Status / Behavior type with START/STOP/POINT), and a time column (Time in seconds, or Image index / Frame in frames).

Parameters:

path – BORIS tabular export (.tsv or .csv).
fps – Frames per second, to convert event times to frames.
n_frames – Length of the raster (match the GameThogram pass).
behaviours – Restrict/order the output columns; defaults to all behaviours seen in the file, sorted.

pyvisor.analysis.reliability.load_gamethogram(path: Path, fps: float, labels: list[str] | None = None) → RasterAnnotation[source]: Load a GameThogram export, dispatching on file suffix.

pyvisor.analysis.reliability.match_event_onsets(onsets_reference: numpy.ndarray, onsets_test: numpy.ndarray, tolerance_frames: int) → PrecisionRecall[source]

Match behaviour onsets within ±*tolerance_frames* (greedy nearest).

Each reference onset may match at most one test onset and vice versa. Returns event-level precision/recall/F1 — the statistic for “annotations land on the right frame”.

pyvisor.analysis.reliability.onset_frames(behaviour_vector: numpy.ndarray) → numpy.ndarray[source]: Frame indices at which the behaviour switches from off to on.

pyvisor.analysis.reliability.percent_agreement(rater_a: numpy.ndarray, rater_b: numpy.ndarray) → float[source]: Raw proportion of frames on which the two raters agree.

pyvisor.analysis.reliability.precision_recall_f1(reference: numpy.ndarray, test: numpy.ndarray) → PrecisionRecall[source]

Precision, recall and F1 of test against reference.

A component is nan when its denominator is zero (e.g. recall when the behaviour never occurs in the reference).

pyvisor.analysis.reliability.proportion_active(raster: numpy.ndarray) → float[source]

Fraction of frames in which the behaviour is active.

With a 1-D behaviour vector this is that behaviour’s index; pass a 2-D (frames × behaviours) slice and the row-wise any to obtain a Courtship Index over a set of courtship behaviours.