API reference
Configuration
Machine-specific path and environment resolution for TadPose.
- exception tadpose.config.PathConfigError[source]
Bases:
RuntimeErrorRaised when machine-specific configuration cannot be resolved.
- tadpose.config.active_profile_name()[source]
Name of the active profile (env → JSON
active_profile→ ‘local’).- Return type:
- tadpose.config.configured_path(key, *fallback_parts)[source]
Profile path
keyif set, elsedata_root()joined with fallbacks.Lets CLI defaults prefer an explicit profile key (e.g.
db_path,output_root,videos_root) so paths need not be typed, while still working on a machine whose profile only setsdata_root(or only setsTADPOSE_DATA_ROOTwith nolocal_paths.jsonat all).
- tadpose.config.data_root()[source]
Resolve the tadpole-project data root (env → JSON → in-repo symlink).
- Return type:
- tadpose.config.export_lines(profile_name=None)[source]
Render the chosen profile as
TADPOSE_<KEY>=valueshell lines.
- tadpose.config.get(key, default=None)[source]
Return a key from the active profile, falling back to
default.
Console entry point that dispatches to the per-stage CLIs.
Acquisition and pose estimation
Finding 24 needles in a Hough-stack.
Detect, correct, and localise the 24 circular wells in a multi-well plate image. Corrects for lens distortion via eigenvector alignment and central-well interpolation.
- class tadpose.well_detection.WellDetector(img_bgr, *, correct_orientation=False, override_radius=None)[source]
Bases:
objectDetect and localise 24 wells in a multi-well plate image.
Runs the full pipeline on construction: Hough circle detection, adaptive parameter search, eigenvector-aligned centre correction, optional orientation flip.
- grey
Greyscale input image.
- centres
(24, 2) corrected well centres, or (0, 2).
- radii
(24,) well radii, or None.
- median_radius
Median well radius in pixels.
- well_separation
Centre-to-centre spacing in pixels.
- top_left
(24, 2) top-left crop corners, or None.
- detection_ok
True if 24 wells were found.
One plate video in, 24 well videos out.
Split a 24-well plate recording into individual per-well greyscale videos. Two-pass approach: Pass 1 Detect well centres in every frame (or just the first), then smooth the centre trajectories to correct for camera drift. Pass 2 Crop each frame at the smoothed centres and write 24 .mp4 files.
- tadpose.video_segmentation.main()[source]
Command-line entry point for plate video splitting.
- Return type:
None
- tadpose.video_segmentation.split_plate_video(video_path, output_dir, *, detect_per_frame=False, frame_limit=0, well_diameter_mm=15.6)[source]
Split a 24-well plate video into 24 individual well videos.
- Parameters:
video_path (Path) – Path to the input .mp4 plate recording.
output_dir (Path) – Directory for the 24 output .mp4 files.
detect_per_frame (bool) – If True, run Hough detection on every frame (slower, better for unstable rigs). If False, detect on the first frame only and reuse.
frame_limit (int) – Max frames to process (0 = all).
well_diameter_mm (float) – Physical well diameter for metadata export.
- Returns:
Path to the output directory.
- Return type:
Probe fps, resolution and timestamp.
Extracts capture metadata (fps, frame size, date-time) from a raw plate video via OpenCV.
- class tadpose.video_info.VideoInfoExtractor(video_path)[source]
Bases:
object- get_video_info()[source]
Return capture metadata (date, time, camera, fps, duration).
Well geometry (
median_well_radius_pixels/real_well_diameter_mm) is deliberately NOT returned here: those belong to the split step (video_segmentation._save_well_metadata), which measures the real radius per plate. Emitting hardcoded 17mm/200px defaults here used to clobber the split’s correct values via write_video_json’s dict merge whenever the launcher re-ran after a resume-gated split – silently corrupting pix2mm.
DeepLabCut video-analysis wrapper.
Thin wrapper that runs DeepLabCut pose estimation over a single per-well clip.
Feature extraction
Turning wiggly pixels into thrust, yaw, and posture.
Extract body-centric velocity (thrust, yaw, slip) and frons-aligned posture from DeepLabCut tracking output.
- tadpose.feature_extraction.align_posture(df, parts=None)[source]
Translate + rotate all landmarks so that the frons sits at the origin and the tail-base lies on the positive x-axis.
Fully vectorised: no Python loops over frames.
- Parameters:
- Returns:
DataFrame with columns (‘{part}_aligned’, ‘x’/’y’) for each part.
- Return type:
- tadpose.feature_extraction.compute_com(df)[source]
Compute the centre of mass proxy (mean of both eyes + tail base).
- tadpose.feature_extraction.compute_frons(df)[source]
Compute the frons (midpoint between the eyes).
- tadpose.feature_extraction.compute_posture_dynamics(aligned, parts=None)[source]
Frame-to-frame difference vectors for aligned posture.
For each landmark, computes dx(t) = x(t) - x(t-1) and likewise for y. The first frame is set to zero.
- tadpose.feature_extraction.compute_velocity(com_xy, yaw, fps=50.0, mm_diameter=15.6, px_diameter=None)[source]
Decompose CoM displacement into thrust, slip, and yaw speed.
All outputs are in physical units (mm/s, rad/s) if px_diameter is provided; otherwise in pixels/frame and rad/frame.
- Parameters:
com_xy (ndarray[tuple[Any, ...], dtype[floating]]) – (N, 2) centre-of-mass positions in pixels.
yaw (ndarray[tuple[Any, ...], dtype[floating]]) – (N,) body orientation angles in radians.
fps (float) – Recording frame rate.
mm_diameter (float) – Physical well diameter in mm.
px_diameter (float | None) – Observed well diameter in pixels. If None, no unit conversion is applied.
- Returns:
Dict with keys ‘thrust’, ‘slip’, ‘yaw_speed’, each (N,). First frame is zero (no previous frame for diff).
- Return type:
- tadpose.feature_extraction.compute_yaw(frons_xy, tail_base_xy)[source]
Body-axis orientation angle (rad) per frame.
- tadpose.feature_extraction.correct_eye_positions(df, threshold=0.5)[source]
Replace low-confidence eye detections with the other eye or the previous frame.
Operates in-place on df. When one eye is below threshold, its position is replaced by the other eye. When both are below threshold, both are forward-filled from the last confident frame.
- Parameters:
- Returns:
The modified DataFrame (same object, modified in-place).
- Return type:
- tadpose.feature_extraction.extract_features(dlc_h5, *, likelihood_threshold=0.5, fps=50.0, well_diameter_mm=15.6, well_diameter_px=None)[source]
Full feature extraction pipeline from a DLC .h5 file.
- Steps:
Load DLC tracking data and drop scorer level.
Correct low-confidence eye detections.
Interpolate low-confidence landmarks.
Compute frons (eye midpoint) and CoM proxy.
Compute body-centric velocity (thrust, yaw, slip).
Align posture (frons at origin, tail-base on x-axis).
Compute posture dynamics (frame-to-frame diffs).
- Parameters:
dlc_h5 (Path) – Path to DeepLabCut .h5 output file.
likelihood_threshold (float) – DLC confidence cutoff.
fps (float) – Recording frame rate.
well_diameter_mm (float) – Physical well diameter for unit conversion.
well_diameter_px (float | None) – Observed well diameter in pixels. If None velocities stay in pixels/frame.
- Returns:
DataFrame combining original tracking, velocity, aligned posture, and posture dynamics columns.
- Return type:
- tadpose.feature_extraction.interpolate_low_confidence(df, body_part, threshold=0.5)[source]
Linearly interpolate positions where DLC likelihood is below threshold. Operates in-place.
- tadpose.feature_extraction.main()[source]
Extract velocity + aligned posture and write them into the DLC .h5.
The pipeline’s extract step: read a per-well DLC tracking file, add the body-centric velocity (
('velocity', …)), frons-aligned posture (('{part}_aligned', …)) and posture-dynamics columns, and write the augmented frame back soresult_managercan ingest it. Mirrors the legacyextract_trajectories.py(--output_path inplaceoverwrites).- Return type:
None
- tadpose.feature_extraction.px_per_frame_to_mm_per_s(values, mm_distance, px_distance, fps=50.0)[source]
Convert pixel-per-frame values to mm/s.
- Parameters:
- Returns:
Array in mm/s.
- Return type:
Scrubbing artefacts from 6×10^7 observations.
Remove tracking artefacts by applying distribution-based thresholds to velocity and posture features. Boundaries derived from logarithmic histogram inspection of rare non- linearities (thesis Appendix A).
- tadpose.feature_cleaning.clean_features(data, boundaries=None)[source]
Remove rows where any feature falls outside its boundary.
- Parameters:
- Returns:
- (cleaned DataFrame with reset index,
list of original row indices that were removed).
- Return type:
- tadpose.feature_cleaning.clean_features_from_array(data, feature_names, boundaries=None)[source]
Clean a numpy array using feature name lookup.
Convenience wrapper for use with .npy clustering matrices.
- tadpose.feature_cleaning.plot_feature_histograms(data, output_path, *, bins=50)[source]
Save a multi-panel figure of log-scale histograms per feature.
Useful for visually identifying artefact tails to set cleaning thresholds.
Z-scoring 10^7 observations without breaking a sweat.
Compute, save, load, and apply z-score normalisation for clustering feature matrices.
- tadpose.normalisation.compute_and_save_mu_sigma(data_path, output_path, *, exclude_columns=None)[source]
Load a .npy or .csv feature matrix, compute mu/sigma, save.
- tadpose.normalisation.compute_mu_sigma(data)[source]
Compute per-column mean and standard deviation.
- tadpose.normalisation.de_z_score(data_z, mu, sigma)[source]
Invert z-score normalisation: x = z * sigma + mu.
- Parameters:
- Returns:
(N, F) matrix in original units.
- Return type:
- tadpose.normalisation.save_mu_sigma(mu, sigma, path)[source]
Write mu and sigma to a two-column CSV.
Clustering
GPU k-means on 10^7 frames, submitted via SLURM.
Partitions z-scored feature vectors into k prototypical behaviours using RAPIDS cuML k-means on GPU. Evaluates cluster quality via Calinski-Harabasz index. Supports contiguous leave-out for stability analysis. Designed for SLURM array-job submission: all parameters (k, deletion size, deletion position, random state) are accepted as CLI arguments. See also: stag.clustering.kmeans (zerotonin/stag) which implements the same pattern for deer gait data.
- tadpose.clustering.calinski_harabasz(labels, data_gpu)[source]
Compute Calinski-Harabasz score, moving data to CPU.
Returns NaN if only one cluster was found.
- Parameters:
labels (np.ndarray)
data_gpu (cp.ndarray)
- Return type:
- tadpose.clustering.leave_out(data, reduction_pct, position_pct)[source]
Remove a contiguous block from data for stability testing.
The block starts at position_pct % through the array and removes reduction_pct % of the rows, wrapping around if the block extends past the end.
- tadpose.clustering.make_output_paths(parent, tag, k, del_size, del_pos)[source]
Generate and create structured output paths.
Directory layout:
{parent}/{tag}/delSize_{del_size}/k_{k}/centroids/ {parent}/{tag}/delSize_{del_size}/k_{k}/labels/ {parent}/{tag}/delSize_{del_size}/k_{k}/
- tadpose.clustering.meta_path(parent, tag, k, del_size, del_pos)[source]
Path to a run’s meta JSON, without creating any directories.
Used by the SLURM submit scripts to check whether a (k, del_size, del_pos) combination has already been clustered.
- tadpose.clustering.run_kmeans(tag, n_clusters, del_size, del_pos, random_state, data_path, save_dir)[source]
Single k-means run with leave-out, designed for SLURM.
- Parameters:
tag (str) – Experiment identifier.
n_clusters (int) – k for k-means.
del_size (int) – Leave-out percentage (0 = full data).
del_pos (int) – Leave-out start position (percentage).
random_state (int) – RNG seed for centroid initialisation.
data_path (Path) – Path to z-scored .npy feature matrix.
save_dir (Path) – Root directory for structured output.
- Return type:
None
Clustering-sweep quality metrics and centroids.
Summarises a Davies-Bouldin / k sweep: stability, quality scores, and de-z-scored centroid metadata.
- class tadpose.cluster_meta.ClusterMetaAnalysis(directory)[source]
Bases:
objectAnalyzes clustering metadata and centroids for stability and patterns.
This class is designed to load clustering metadata and centroid information from JSON files, calculate stability metrics across clustering attempts, and visualize results.
- df
DataFrame holding loaded clustering metadata.
- Type:
pd.DataFrame
- analyze()[source]
Performs the analysis by loading data, calculating instability, and optionally visualizing results.
- calculate_and_assign_instability()[source]
Calculates and assigns instability values to the DataFrame for each clustering attempt.
- static calculate_instability(centroids_list)[source]
Calculates the instability measure for a list of centroids arrays.
Parameters: - centroids_list: List of numpy arrays, each containing the centroids of one clustering attempt.
Returns: - Instability measure for each clustering attempt.
- de_zscore_centroids(centroids, mu, sigma)[source]
Reverses the z-scoring process for centroids using the original mean (mu) and standard deviation (sigma).
- find_most_stable_centroids(k_number, reduction_percent)[source]
Returns the most stable centroids for a given k_number and reduction_percent, identified by the minimal instability. If there’s a tie, the first one is taken.
Parameters: - k_number: The number of clusters. - reduction_percent: The reduction percentage.
Returns: - The centroids array of the most stable clustering attempt.
- load_centroids_for_analysis(reduction_percent, k_number)[source]
Loads centroids for a specific reduction_percent and k_number.
Parameters: - reduction_percent: The reduction percentage of interest. - k_number: The number of clusters (k) of interest.
Returns: - List of centroids arrays for the specified reduction_percent and k_number.
- load_data()[source]
Loads clustering metadata from JSON files within the specified directory.
- Returns:
A DataFrame containing the clustering metadata.
- Return type:
pd.DataFrame
- load_df(load_path)[source]
Loads the DataFrame from a CSV file at the specified path.
- Parameters:
load_path (str) – The path from where the DataFrame should be loaded.
- Returns:
The loaded DataFrame.
- Return type:
pd.DataFrame
- normalize_centroids(centroids)[source]
Normalizes the centroids in each feature to the absolute max value of that feature across all centroids.
Parameters: - centroids: A 2D NumPy array of centroids, shape (n_centroids, n_features)
Returns: - Normalized centroids: A 2D NumPy array, shape (n_centroids, n_features)
- tadpose.cluster_meta.main()[source]
CLI: summarise a clustering sweep and save the quality-metric figures.
- Return type:
None
- tadpose.cluster_meta.write_results_to_jsonFile(feature_labels, centroids, centroids_uf, centroids_nmax, json_filepath)[source]
Align cluster centroids across runs.
Attaches centroid labels to feature rows and computes the average posture before/after a given cluster split.
- class tadpose.centroid_matching.CentroidProcessor(label_file, data_file, centroids_file)[source]
Bases:
object- calculate_95th_percentile_for_velocity_features()[source]
Calculate the 95th percentile for thrust, slip, and yaw maximum values.
Returns: - percentiles (dict): A dictionary with keys ‘thrust_mm_s’, ‘slip_mm_s’, ‘yaw_rad_s’ and their corresponding 95th percentiles.
- get_data_with_and_without_cluster_k(k)[source]
This method returns two NumPy arrays: 1. Data points corresponding to the cluster label ‘k’. 2. Data points not corresponding to the cluster label ‘k’.
Args: - k (int): The cluster label to filter data by.
Returns: - data_with_k (np.ndarray): Data points corresponding to cluster label ‘k’. - data_without_k (np.ndarray): Data points not corresponding to cluster label ‘k’.
- inverse_rotate_data(rotated_data, shift)[source]
Reverses the rotation applied by rotate_data by shifting the data back.
Parameters: - rotated_data (np.ndarray): The rotated data array. - shift (int): The number of positions the data was originally shifted.
Returns: - original_data (np.ndarray): The data array rotated back to its original state.
- sample_cluster_indices()[source]
Pick one representative index per cluster label.
- Returns:
A tuple
(sampled_indices, sampled_data).sampled_indicesholds one index per cluster label;sampled_dataholds a dict per sampled index withbody_position_before,body_position_afterandvelocity_and_diffsentries.
Cluster validation (choosing k)
Internal cluster-validation metrics: Silhouette, Inertia, Kneedle elbow.
- tadpose.analysis.internal_metrics.compute_inertia(X, centroids, labels, chunk_size=1000000, columns=None)[source]
Within-cluster sum of squared distances — the k-means objective W(k).
Streams over the feature matrix in
chunk_sizerows so the residual array never exceedschunk_size × n_features × 8bytes of working memory, rather than materialising the fullX - centroids[labels].- Parameters:
X (ndarray) – Feature matrix,
(n_samples, n_features).centroids (ndarray) – Centroid matrix,
(n_clusters, n_features).labels (ndarray) – Per-sample cluster assignment,
(n_samples,).chunk_size (int) – Rows per streaming chunk.
columns (Sequence[int] | None) – Optional column subset to select per chunk so that X matches the feature space the clustering used (e.g. the 16-column velocity + posture-diff subset of a wider feature matrix). Selection happens per chunk so memory stays bounded.
- Returns:
Scalar inertia, matching
KMeans.inertia_for a converged fit.- Return type:
- tadpose.analysis.internal_metrics.compute_instability(centroids_list)[source]
Per-attempt instability for repeated clusterings at one (k, reduction).
For each pair of clustering attempts the centroid sets are matched optimally (Hungarian assignment on the Euclidean cost matrix) and the matched distance summed. The most stable attempt is the one with the smallest total distance to all others; each attempt’s instability is its matched distance to that reference attempt. Lower is more stable.
- tadpose.analysis.internal_metrics.compute_silhouette_stratified(X, labels, n_per_cluster=5000, n_repeats=50, rng=None, columns=None)[source]
Mean silhouette score via stratified per-cluster subsampling.
Silhouette is \(\mathcal{O}(n^2)\) in pairwise distances, so on the full ~$10^{7}$ tadpole sample it is infeasible. This helper draws
n_per_clustersamples from every cluster, computes silhouette on that subsample, and repeatsn_repeatstimes to give a median ± IQR.Stratification matters: the rare seizure motifs (some C-shaped contraction and impact-compression clusters are <1 % of the data) would be under-represented by uniform subsampling (
silhouette_score(..., sample_size=N)), biasing the metric.- Parameters:
X (ndarray) – Feature matrix,
(n_samples, n_features).labels (ndarray) – Per-sample cluster assignment.
n_per_cluster (int) – Samples per cluster per repeat.
n_repeats (int) – Number of independent subsamples.
rng (Generator | None) – NumPy generator; default-seeded if None.
columns (Sequence[int] | None) – Optional column subset (applied to each subsample) so X matches the clustered feature space.
- Returns:
Dict with
mean_silhouette(median over repeats),iqr_silhouette((lower, upper) quartiles),per_repeat(1-D array), andper_cluster_mean(median silhouette per cluster).- Return type:
- tadpose.analysis.internal_metrics.gather_meta_metrics(meta_dir, reductions=None)[source]
Walk clustering metadata JSONs for the cheap (meta-only) metrics.
Reads each fit’s Calinski-Harabasz score and centroids (no feature matrix needed), and computes
compute_instability()per(k, reduction_percent)group. This is the fast half of the selection sweep; inertia and silhouette (which need the feature matrix) are added separately.
- tadpose.analysis.internal_metrics.locate_elbow_kneedle(k_values, inertia, S=2.0, curve='convex', direction='decreasing')[source]
Locate the elbow on a W(k) curve via the Kneedle algorithm.
Wraps
kneed.KneeLocator(Satopää et al. 2011) and returns the chosen k plus diagnostic fields.- Parameters:
k_values (Sequence[int]) – Monotonic increasing sequence of cluster counts.
inertia (Sequence[float]) –
W(k)values in the same order (non-increasing).S (float) – Sensitivity (lower → more aggressive detection).
curve (str) –
"convex"for inertia curves.direction (str) –
"decreasing"for inertia curves.
- Returns:
Dict with
elbow_k(or None),elbow_y,elbow_indexandnormalised_knee_distance.- Return type:
- tadpose.analysis.internal_metrics.main()[source]
CLI: build a k-selection summary (CH, inertia, Kneedle, silhouette).
- Return type:
None
- tadpose.analysis.internal_metrics.parse_columns(spec)[source]
Parse a column spec like
"0,1,2,16-28"into a list of indices.Needed when the clustering used a subset of a wider feature matrix (e.g. the published sweep clustered velocity + posture-diff columns
0,1,2,16-28out of a 29-column matrix).
- tadpose.analysis.internal_metrics.plot_selection(summary, elbow_k, output_path)[source]
Plot CH, silhouette and inertia/elbow versus k.
Renders three panels (Calinski-Harabasz, mean silhouette, inertia with the Kneedle elbow marked) and saves them through
tadpose.viz_constants.save_figure()(editable-text SVG + PNG + CSV).- Parameters:
summary (DataFrame) – A
selection_summary()DataFrame.elbow_k (int | None) – k flagged by
locate_elbow_kneedle(), or None.output_path – Base path (no extension) for the saved figure.
- tadpose.analysis.internal_metrics.plot_selection_metrics(summary, *, chosen_k, output_path, title=None, elbow_k=None, null=None)[source]
Render the 4-panel cluster-selection figure (one metric per panel).
Each panel plots a metric versus
k; metrics computed across severalreduction_percentlevels are drawn as a sequential-colour sweep with a shaded inter-quartile band, those computed at a single level as one emphasised line. The chosenkis marked, the Kneedle elbow starred on the inertia panel, and an optionalnullband shaded in grey.- Parameters:
summary (DataFrame) – Long per-
(k, reduction_percent)table with median columns (silhouette/calinski_harabasz/inertia/instability) and optional*_low/*_highbounds.chosen_k (int) – The selected number of clusters (vertical marker).
output_path – Base path (no extension); saved via
save_figure.title (str | None) – Optional figure suptitle.
elbow_k (int | None) – Kneedle elbow on the inertia panel, if any.
null (DataFrame | None) – Optional per-
knull-model band (columnskplus<metric>_low/<metric>_high).
- tadpose.analysis.internal_metrics.recompute_inertia_for_meta_dir(meta_dir, data_path, overwrite=False, workers=1, reduction_percents=None, feature_columns=None)[source]
Back-fill
inertiafor every metadata JSON undermeta_dir.Historical runs saved Calinski-Harabasz but not
inertia; the Kneedle elbow needsW(k). This walks the JSON tree, locates the matching centroids/labels, recomputesW(k), and optionally writes it back into each JSON.- Parameters:
meta_dir (str | Path) – Root directory of the metadata JSON files.
overwrite (bool) – When True, add an
"inertia"field to each JSON in place.workers (int) – Process-pool size (workers share the matrix via
mmap_mode="r").reduction_percents (Sequence[float] | None) – Optional whitelist of
reduction_percentvalues; others are skipped.
- Returns:
DataFrame with
file_path,k_number,reduction_percent,cut_position_percentandinertia.- Return type:
- tadpose.analysis.internal_metrics.selection_summary(k_values, ch=None, ch_low=None, ch_high=None, instability=None, instability_low=None, instability_high=None, silhouette=None, silhouette_low=None, silhouette_high=None, inertia=None, inertia_low=None, inertia_high=None)[source]
Assemble a per-k DataFrame of selection metrics (medians + bounds).
Each metric may carry optional
_low/_highbounds aligned tok_values(e.g. quartiles) for a confidence band; missing bounds appear as NaN columns.- Returns:
DataFrame with one row per k and median/low/high columns for Calinski-Harabasz, instability, silhouette and inertia.
- Parameters:
- Return type:
Post-clustering analysis
How much time does each tadpole spend in each cluster?
Compute cluster membership proportions at the trial level, aggregate by experimental group, and prepare long-format DataFrames for statistical testing and visualisation.
- tadpose.analysis.proportions.analyse_proportions(csv_paths, categories, label_col='cluster_label', trial_col='trial_id', n_clusters=None, *, output_dir=None)[source]
Full proportion analysis pipeline.
Load and concatenate CSVs.
Assign experimental groups.
Compute per-trial proportions.
Melt to long format.
Summarise (mean ± SEM per group per cluster).
- Parameters:
csv_paths (list[Path]) – List of CSV files containing trial data with cluster labels.
categories (dict[str, list[int]]) – Group definitions: {group_name: [trial_ids]}.
label_col (str) – Cluster label column in the CSVs.
trial_col (str) – Trial ID column.
n_clusters (int | None) – Total k (inferred if None).
output_dir (Path | None) – If set, save intermediate CSVs here.
- Returns:
(wide_proportions, long_proportions, group_summary_df)
- Return type:
- tadpose.analysis.proportions.assign_groups(df, categories, trial_col='trial_id')[source]
Add a ‘group’ column based on a dict mapping group names to lists of trial IDs.
- tadpose.analysis.proportions.compute_trial_proportions(df, trial_col='trial_id', label_col='cluster_label', n_clusters=None)[source]
Compute the proportion of frames assigned to each cluster for every trial.
- Parameters:
- Returns:
one row per trial, one column per cluster (named ‘cluster_0’, ‘cluster_1’, …). Values are proportions (0–1).
- Return type:
Wide-format DataFrame
- tadpose.analysis.proportions.group_summary(long, group_col='group', cluster_col='cluster', value_col='proportion')[source]
Compute mean, SEM, and n per group per cluster.
- tadpose.analysis.proportions.proportions_long(wide, trial_col='trial_id', group_col=None, group_map=None)[source]
Melt wide-format proportions to long format.
- Parameters:
wide (DataFrame) – Output of compute_trial_proportions().
trial_col (str) – Trial ID column.
group_col (str | None) – If present in wide, kept as-is. Otherwise generated from group_map.
group_map (dict[int, str] | None) – Dict mapping trial_id → group name. Used to add a group column if group_col is None.
- Returns:
trial_id, cluster, proportion, and optionally group.
- Return type:
Long DataFrame with columns
Letting reRandomStats do the heavy lifting.
Thin wrappers around rerandomstats.MultiGroupTest and rerandomstats.FisherResamplingTest for comparing cluster proportions across experimental groups. Replaces the hand-rolled Shapiro → Kruskal → Mann-Whitney → Bonferroni chains that were duplicated across 6 scripts in the original analysis/ directory. Dependency: pip install rerandomstats (github.com/zerotonin/reRandomStats).
- tadpose.analysis.stats.compare_cluster_proportions(proportions, group_col='group', cluster_col='cluster', value_col='proportion', *, alpha=0.05, output_dir=None)[source]
Run group comparisons independently for each cluster.
For each unique value in cluster_col, subsets the data and runs compare_groups(). Results are concatenated into a single DataFrame with an additional ‘cluster’ column.
- Parameters:
proportions (DataFrame) – Long-format DataFrame with columns for group, cluster, and proportion value.
group_col (str) – Column identifying experimental group.
cluster_col (str) – Column identifying the cluster.
value_col (str) – Column with the proportion values.
alpha (float) – Significance level.
output_dir (Path | None) – If set, save per-cluster CSV results here.
- Returns:
Concatenated DataFrame of all pairwise results.
- Return type:
- tadpose.analysis.stats.compare_groups(df, value_col, group_col, *, test='Fisher:meanDiff', combination_n=10000, correction_type='fdr_bh')[source]
Compare a continuous measure across experimental groups.
Runs pairwise Fisher resampling (permutation) tests across every group pair via
rerandomstats.MultiGroupTest, correcting the p-values with Benjamini-Hochberg FDR by default.- Parameters:
df (DataFrame) – DataFrame with at least value_col and group_col.
value_col (str) – Continuous measurement (e.g. cluster proportion).
group_col (str) – Experimental-group identifier.
test (str) – reRandomStats
family:namespec; default Fisher resampling on the mean difference.combination_n (int | str) – Permutations per comparison (
"all"for exact).correction_type (str) – Multiple-comparison correction (default BH-FDR).
- Returns:
groupA,groupB, their n,p value,p value corrected,handsig. level.- Return type:
DataFrame with one row per group pair
- tadpose.analysis.stats.permutation_test_proportions(group_a, group_b, *, func='meanDiff', n_resamples=10000, alpha=0.05)[source]
Fisher resampling (permutation) test for a difference between groups.
Uses
rerandomstats.FisherResamplingTest;main()returns the two-sided permutation p-value.- Parameters:
- Returns:
Dict with
observed_diff,p_valueandsignificant.- Return type:
Giving 36 arbitrary numbers a name and an order.
K-means cluster IDs are arbitrary. This module provides a mapping from raw cluster numbers to behavioural categories and a canonical display order for publication figures. The mapping is stored as a JSON file so Alex and Bart can edit it without touching Python code. A default mapping is built in for the k=36 clustering used in the thesis.
- class tadpose.analysis.cluster_map.ClusterInfo(raw_id, display_id, category, short_label='', description='')[source]
Bases:
objectMetadata for a single behavioural prototype.
- class tadpose.analysis.cluster_map.ClusterMap(k, entries=<factory>, _by_raw=<factory>)[source]
Bases:
objectFull mapping for one clustering solution.
- Parameters:
k (int)
entries (list[ClusterInfo])
_by_raw (dict[int, ClusterInfo])
- entries
List of ClusterInfo, one per cluster.
- _by_raw
Lookup dict raw_id → ClusterInfo (built lazily).
- entries: list[ClusterInfo]
- classmethod load(path)[source]
Load a mapping from a JSON file.
- Parameters:
path (Path)
- Return type:
- tadpose.analysis.cluster_map.identity_map(k, default_category='unclassified')[source]
Create a trivial 1:1 mapping where display_id == raw_id.
Useful as a starting point before manual annotation.
- Parameters:
- Return type:
- tadpose.analysis.cluster_map.map_from_dict(raw_to_category, sort_key='category')[source]
Build a ClusterMap from a simple {raw_id: category} dict.
Display IDs are assigned by sorting on sort_key (either ‘category’ for grouping by behaviour, or ‘raw_id’ for original order).
- Parameters:
- Returns:
ClusterMap with display IDs assigned.
- Return type:
- tadpose.analysis.cluster_map.thesis_k36_map()[source]
The canonical k=36 mapping with GROUP.index short labels.
Built from the ten behavioural groups (thesis Fig 3.13 cluster_sets) and the prevalence ordering encoded in
tadpose.viz_constants.THESIS_K36_GROUPS, so every prototype carries a label such asCSC.1instead of a bare k-means id. Display IDs follow the group order then within-group prevalence.- Return type:
Transition matrices and preferred-transition tests.
Core Markov chain machinery: empirical transition matrix, a-priori and global-null significance, and the transition figures.
- class tadpose.analysis.markov_chain.TadpoleMarkovChain(csv_file, label_column='agglom_3', npy_file='transition_matrix.npy', usecols=None)[source]
Bases:
object
- tadpose.analysis.markov_chain.main()[source]
CLI: build an Markov chain from a label array and save the transition figures.
- Return type:
None
- tadpose.analysis.markov_chain.plot_above_transitions(markov_chain, ax=None)[source]
Plots a directed graph showing only significant above-average transitions with color-coded nodes and labels inside.
- Parameters:
markov_chain (TadpoleMarkovChain) – An instance of the TadpoleMarkovChain class.
ax (matplotlib.axes.Axes, optional) – The axes to plot on. Defaults to None.
- tadpose.analysis.markov_chain.plot_threshold_transitions(markov_chain, ax=None, threshold=0.14)[source]
Plots a directed graph showing transitions above a given threshold, with color-coded nodes and labels inside.
- Parameters:
markov_chain (TadpoleMarkovChain) – An instance of the TadpoleMarkovChain class.
ax (matplotlib.axes.Axes, optional) – The axes to plot on. Defaults to None.
threshold (float, optional) – The minimum transition probability to include an edge. Defaults to 0.20.
- tadpose.analysis.markov_chain.plot_transition_matrix(markov_chain, ax=None, cmap='viridis', annot=True)[source]
Plots the transition matrix as a heatmap with significance markers.
- Parameters:
markov_chain (TadpoleMarkovChain) – An instance of the TadpoleMarkovChain class.
ax (matplotlib.axes.Axes, optional) – The axes to plot on. Defaults to None.
cmap (str, optional) – The colormap to use for the heatmap. Defaults to ‘viridis’.
annot (bool, optional) – Whether to annotate the heatmap with transition probabilities. Defaults to True.
Grouped Markov chain comparison across conditions.
Builds and plots transition matrices per experimental group (PTZ dose series, 4-AP, neurod2) for biological comparison.
- class tadpose.analysis.markov_chain_groups.TadpoleMarkovChain(df, label_column='agglom_7', npy_file='transition_matrix.npy')[source]
Bases:
object
- class tadpose.analysis.markov_chain_groups.TadpoleMarkovChainGroupAnalysis(csv_file, label_column='agglom_7', npy_file_base='transition_matrix')[source]
Bases:
object- plot_and_save_chains(group_categories, output_dir_base)[source]
Processes each category in group_categories, creates separate folders, and saves all associated Markov chain plots inside their respective folders.
Parameters: - group_categories (dict): Dictionary containing categories with their criteria and labels. - output_dir_base (str): Base directory where category folders will be created.
- tadpose.analysis.markov_chain_groups.main()[source]
CLI: run grouped Markov chain comparisons and save the per-category figures.
- Return type:
None
- tadpose.analysis.markov_chain_groups.plot_threshold_transitions(markov_chain, ax=None, threshold=0.14)[source]
- tadpose.analysis.markov_chain_groups.plot_transition_matrix(markov_chain, ax=None, cmap='viridis', annot=True)[source]
Add new recordings to an existing clustering by nearest centroid.
The careful part is normalisation: the new features must be z-scored with the
original clustering’s saved mu and sigma, never statistics recomputed from
the new data. normalise_and_assign() enforces that by taking mu/sigma as
arguments (load them with tadpose.normalisation.load_mu_sigma()); the
legacy assign_clusters_from_numpy() assumes already-z-scored input and is
kept for backward compatibility.
- tadpose.analysis.assign_new_data_to_clusters.assign_clusters_from_numpy(json_input, numpy_input, output_file)[source]
- tadpose.analysis.assign_new_data_to_clusters.assign_new_data_to_clustering(raw_feature_path, mu_sigma_path, centroids_path, output_label_path, *, feature_columns=None, append_to=None, chunk_size=2000000)[source]
File orchestrator: load, z-score with original mu/sigma, assign, save.
Streams the raw features in blocks (memory-bounded), so it handles the full multi-million-row feature matrices without materialising a z-scored copy.
- Parameters:
raw_feature_path (Path) –
.npyof the new, un-normalised features.mu_sigma_path (Path) – the clustering’s saved mu/sigma CSV.
centroids_path (Path) – the clustering’s centroids (
.npyor.json).output_label_path (Path) – destination
.npyfor the new labels.feature_columns (Sequence[int] | None) – clustering feature columns (see
normalise_and_assign()).append_to (Path | None) – if given, an existing labels
.npyfrom the SAME clustering; the new labels are concatenated onto it and written tooutput_label_path.chunk_size (int)
- Return type:
- tadpose.analysis.assign_new_data_to_clusters.main()[source]
CLI: assign new feature rows to the nearest cluster centroid.
With
--mu-sigmathis runs the safe path: the raw features are z-scored with the clustering’s saved statistics before assignment. Without it, the legacy behaviour (already-z-scored input, JSON centroids) is used. Paths default underconfig.data_root()so nothing machine-specific is hardcoded.- Return type:
None
- tadpose.analysis.assign_new_data_to_clusters.nearest_centroid(z, centroids, *, chunk_size=2000000)[source]
Nearest-centroid (Euclidean) label per row, chunked for large inputs.
Rows with any non-finite value are left unassigned (label
-1) rather than collapsing to centroid 0. Distances use|c|^2 - 2 x.c(the|x|^2term is constant per row and irrelevant to the argmin).
- tadpose.analysis.assign_new_data_to_clusters.normalise_and_assign(raw_features, mu, sigma, centroids, *, feature_columns=None, chunk_size=2000000)[source]
z-score new data with the ORIGINAL mu/sigma, then assign to nearest centroid.
- Parameters:
raw_features (ndarray[tuple[Any, ...], dtype[floating]]) – (N, F) un-normalised new feature matrix.
mu (ndarray[tuple[Any, ...], dtype[floating]]) – (F,) the clustering’s SAVED statistics (load with
tadpose.normalisation.load_mu_sigma()). Never recompute these from the new data.sigma (ndarray[tuple[Any, ...], dtype[floating]]) – (F,) the clustering’s SAVED statistics (load with
tadpose.normalisation.load_mu_sigma()). Never recompute these from the new data.centroids (ndarray[tuple[Any, ...], dtype[floating]]) – (K, len(feature_columns)) centroids in z-scored space.
feature_columns (Sequence[int] | None) – Column indices the clustering used (e.g.
[0, 1, 2, *range(16, 29)]for posture+velocity);Noneuses all columns.chunk_size (int) – Rows processed per block.
- Returns:
(N,) int32 labels;
-1where a row had a non-finite clustering feature.- Return type:
Fold raw cluster ids into agglomerated categories.
Maps per-frame cluster labels onto agglomerated behavioural categories defined by cluster-map JSON files.
- tadpose.analysis.generate_new_labelling.add_clustering_labels(npy_filepath, json_filepaths, output_filepath, new_label_column_names)[source]
Load a .npy file into a pandas DataFrame, add new label columns based on multiple JSON files, and save the DataFrame as a CSV file.
Parameters: npy_filepath (str): Path to the .npy file containing time_series_id, tadpole_id, and label. json_filepaths (list): List of paths to JSON files containing the clustering label mappings. output_filepath (str): Path to save the resulting CSV file. new_label_column_names (list): List of names for the new columns to be added to the DataFrame.
Visualisation
One source of truth for colours, paths, and figure rules.
Central configuration for all TadPose visualisations. Import this module instead of hardcoding hex values. Wong (2011) colourblind-safe palette with semantic mappings to behavioural categories identified in the clustering.
- tadpose.viz_constants.apply_tadpose_style()[source]
Set TadPose matplotlib defaults globally.
Call at the top of any plotting script or notebook.
- Return type:
None
- tadpose.viz_constants.kinematic_label(raw_id)[source]
Return the KIN.<rank> label for a kinematics k=8 raw id.
- tadpose.viz_constants.pm_category(raw_id)[source]
Return the behavioural-group key for a raw cluster id.
- tadpose.viz_constants.pm_label(raw_id)[source]
Return the GROUP.index label for a raw cluster id (fallback
C<id>).
- tadpose.viz_constants.save_figure(fig, path, *, formats=('svg', 'png'), dpi=300, csv_data=None)[source]
Save a figure as SVG and PNG, with optional CSV data export.
SVG output uses editable text (not curves). Both formats are saved side-by-side in the same directory.
- Parameters:
fig (plt.Figure) – Matplotlib Figure to save.
path (Path) – Base path (without extension). Extensions are added automatically.
formats (tuple[str, ...]) – Tuple of format strings (default: svg + png).
dpi (int) – Resolution for raster formats.
csv_data (Optional[dict[str, object]]) – If provided, a dict of DataFrames/arrays to save as CSVs alongside the figure. Keys become filenames:
{path.stem}_{key}.csv.
- Returns:
List of all saved file paths.
- Return type:
list[Path]
- tadpose.viz_constants.sig_letter(p)[source]
Convert a p-value to letter notation for compact tables.
Posture silhouettes and kinematics crosses.
Publication-quality centroid visualisation combining:.
- tadpose.analysis.viz_centroids.plot_centroid(positions, dynamics, thrust, slip, yaw, *, cluster_id=None, max_thrust=1.0, max_slip=1.0, max_yaw=1.0, max_lim=None, show_scale_bar=True, layout='side', figsize=None)[source]
Create a combined posture + kinematics figure for one centroid.
- Parameters:
positions (dict[str, tuple[float, float]]) – Frons-aligned body-part positions.
dynamics (dict[str, tuple[float, float]] | None) – Posture dynamics vectors (dx, dy per part).
thrust (float) – Kinematic values for this centroid.
slip (float) – Kinematic values for this centroid.
yaw (float) – Kinematic values for this centroid.
cluster_id (int | None) – Optional cluster number for the title.
max_thrust/slip/yaw – Global maxima for normalisation.
max_lim (float | None) – Posture axis limit (use same for all clusters).
show_scale_bar (bool) – Draw scale bar on posture panel.
layout (str) – ‘side’ = kinematics right of posture, ‘below’ = kinematics below frons.
figsize (tuple[float, float] | None) – Override figure size.
max_thrust (float)
max_slip (float)
max_yaw (float)
- Returns:
matplotlib Figure.
- Return type:
Figure
- tadpose.analysis.viz_centroids.plot_centroid_grid(all_positions, all_dynamics, all_thrust, all_slip, all_yaw, *, ncols=6, figsize_per_cell=(2.2, 1.5), output_path=None)[source]
Grid of centroid panels, one per cluster.
Only the bottom-left cell shows the scale bar to avoid clutter.
- Parameters:
all_positions (list[dict[str, tuple[float, float]]]) – List of position dicts, one per cluster.
all_dynamics (list[dict[str, tuple[float, float]] | None]) – List of dynamics dicts (or None).
all_thrust/slip/yaw – Arrays of kinematic values, one per cluster.
ncols (int) – Columns in the grid.
figsize_per_cell (tuple[float, float]) – Size of each sub-panel.
output_path (Path | None) – If set, save SVG + PNG + data CSV.
- Returns:
matplotlib Figure.
- Return type:
Figure
- tadpose.analysis.viz_centroids.plot_kinematics_cross(ax, thrust, slip, yaw, *, max_thrust=1.0, max_slip=1.0, max_yaw=1.0, cross_radius=0.9)[source]
Draw the body-centric kinematics cross.
Thrust points UP, slip points LEFT (positive) / RIGHT (negative), yaw is a half-circle arc above the cross that acts as an arrow. All components are normalised to their respective global maxima so arrows are comparable across clusters.
- Parameters:
ax (Axes) – Axes to draw on (will be set to equal aspect).
thrust (float) – Thrust value (positive = forward).
slip (float) – Slip value (positive = left).
yaw (float) – Yaw value (positive = left/CCW).
max_thrust (float) – Global maximum absolute thrust for normalisation.
max_slip (float) – Global maximum absolute slip for normalisation.
max_yaw (float) – Global maximum absolute yaw for normalisation.
cross_radius (float) – Size of the reference cross arms.
- Return type:
None
- tadpose.analysis.viz_centroids.plot_posture(ax, positions, dynamics=None, *, show_scale_bar=True, scale_bar_length=5.0, scale_bar_unit='px', max_lim=None)[source]
Draw a tadpole posture with optional movement vectors.
- Parameters:
ax (Axes) – Axes to draw on.
positions (dict[str, tuple[float, float]]) – Dict mapping body-part name to (x, y) position in the frons-aligned coordinate frame.
dynamics (dict[str, tuple[float, float]] | None) – Dict mapping body-part name to (dx, dy) posture dynamics vectors. If None, vectors are omitted.
show_scale_bar (bool) – Draw an xy scale bar in the bottom-left.
scale_bar_length (float) – Length of the scale bar in data units.
scale_bar_unit (str) – Unit label for the scale bar.
max_lim (float | None) – If set, axes are clipped to ±max_lim on both axes (use the same value across all clusters for visual consistency).
- Return type:
None
Rainclouds, not bar charts.
Publication-quality visualisations for comparing cluster proportions and velocity measures across experimental groups. Primary plot type: raincloud (half-violin + jittered strip + boxplot summary) with optional significance brackets. Uses Wong (2011) palette from viz_constants. All figures exported as SVG (editable text) + PNG + CSV data.
- tadpose.analysis.viz_proportions.proportion_bars(summary, *, group_col='group', cluster_col='cluster', mean_col='mean', sem_col='sem', group_order=None, cluster_order=None, colours=None, figsize=None, output_path=None)[source]
Grouped bar chart of mean proportions with SEM error bars.
- Parameters:
summary (DataFrame) – Output of proportions.group_summary().
group_col (str) – Group column.
cluster_col (str) – Cluster column.
mean_col (str) – Mean proportion column.
sem_col (str) – SEM column.
cluster_order (list[int] | None) – Display order of clusters.
output_path (Path | None) – If set, save SVG + PNG + CSV.
- Returns:
matplotlib Figure.
- Return type:
Figure
- tadpose.analysis.viz_proportions.raincloud(ax, data, value_col, group_col, *, group_order=None, colours=None, violin_side='left', show_box=True, ylabel='Proportion', title='')[source]
Draw a raincloud plot on ax.
- Parameters:
ax (Axes) – Axes to draw on.
data (DataFrame) – Long-format DataFrame.
value_col (str) – Column with continuous values.
group_col (str) – Column with group labels.
group_order (list[str] | None) – Display order of groups. If None, sorted alpha.
colours (dict[str, str] | None) – Dict mapping group name to hex colour. Defaults to GROUP_COLOURS.
violin_side (str) – ‘left’ (cloud left, rain right) or ‘right’.
show_box (bool) – Draw mini box plots.
ylabel (str) – Y-axis label.
title (str) – Axes title.
- Return type:
None
- tadpose.analysis.viz_proportions.raincloud_grid(long_df, stats_df=None, *, value_col='proportion', group_col='group', cluster_col='cluster', cluster_order=None, group_order=None, colours=None, ncols=6, cell_size=(3.0, 2.5), output_path=None)[source]
Grid of raincloud plots, one per cluster.
- Parameters:
long_df (DataFrame) – Long-format proportions (from proportions_long).
stats_df (DataFrame | None) – Output of stats.compare_cluster_proportions(). Must have a ‘cluster’ column.
value_col (str) – Column with proportion values.
group_col (str) – Column with group labels.
cluster_col (str) – Column with cluster IDs.
cluster_order (list[int] | None) – Display order of clusters (use ClusterMap.display_order()).
ncols (int) – Grid columns.
cell_size (tuple[float, float]) – (width, height) per sub-panel.
output_path (Path | None) – If set, save SVG + PNG + CSV.
- Returns:
matplotlib Figure.
- Return type:
Figure
- tadpose.analysis.viz_proportions.raincloud_with_stats(data, value_col, group_col, stats_df=None, *, group_order=None, colours=None, ylabel='Proportion', title='', figsize=(4, 3.5), alpha=0.05)[source]
Raincloud plot with optional significance brackets.
If stats_df is provided (output of analysis.stats.compare_groups), significant pairs are annotated with star brackets.
- Parameters:
data (DataFrame) – Long-format DataFrame.
value_col (str) – Column with values.
group_col (str) – Column with group labels.
stats_df (DataFrame | None) – DataFrame with columns ‘group_1’, ‘group_2’, ‘p_adjusted’ (or ‘p_value’), ‘significant’. Output of stats.compare_groups().
ylabel (str) – Y-axis label.
title (str) – Figure title.
alpha (float) – Significance cutoff for brackets.
- Returns:
matplotlib Figure.
- Return type:
Figure
Database and experiment management
Relational schema for 10^7 tadpole observations.
SQLAlchemy ORM models for the tadpole behavioural database. Schema mirrors the experimental hierarchy: ExperimentType → ExperimentSeries → Video → Trial ↑ ↓ Frog → TadpoleGroup ────────────────┘ TimeSeries ↓ ↓ ↓ Trajectory Posture Velocity ↓ Clustering.
- class tadpose.database.Base(**kwargs)[source]
Bases:
DeclarativeBase- Parameters:
kwargs (Any)
- metadata: ClassVar[MetaData] = MetaData()
Refers to the
_schema.MetaDatacollection that will be used for new_schema.Tableobjects.See also
orm_declarative_metadata
- registry: ClassVar[_RegistryType] = <sqlalchemy.orm.decl_api.registry object>
Refers to the
_orm.registryin use where new_orm.Mapperobjects will be associated.
- class tadpose.database.BodyPart(**kwargs)[source]
Bases:
BaseAnatomical landmark tracked by DeepLabCut.
- body_marker
- body_part_id
- postures
- trajectories
- class tadpose.database.Clustering(**kwargs)[source]
Bases:
BaseCluster assignment for one frame under one clustering config.
- centroid
- clustering_id
- clustering_type
- clustering_type_id
- time_series
- time_series_id
- class tadpose.database.ClusteringFeatureStat(**kwargs)[source]
Bases:
BaseThe z-score μ/σ of one feature under one run — stored, not a file path.
New data (genetic edits) are normalised with these values before nearest- centroid assignment, so the normalisation travels with the run in the DB.
- clustering_feature_stat_id
- clustering_run
- clustering_run_id
- feature_index
- feature_name
- mu
- sigma
- class tadpose.database.ClusteringRun(**kwargs)[source]
Bases:
BaseOne stored clustering solution plus its full provenance.
A run fixes the feature set, k, distance, init and seed. The z-score μ/σ live relationally in
ClusteringFeatureStat, and which trials were fit (direct) vs projected (assigned) inClusteringRunTrial— so every label is fully reproducible from the DB alone. Multiple runs coexist (posture-diff+velocity at one k, velocity-only at another, candidate k).- clustering_run_id
- code_version
- created
- distance
- feature_set
- feature_stats
- frame_clusters
- init
- k
- n_features
- name
- notes
- proportions
- run_trials
- seed
- class tadpose.database.ClusteringRunTrial(**kwargs)[source]
Bases:
BasePer-trial membership of a run: was this animal clustered or projected?
One row per (run, trial).
assignmentis a property of the whole trial, not the frame, so it lives here — never duplicated across the trial’s millions ofFrameClusterrows. A frame’s provenance isframe_cluster → time_series → trial → clustering_run_trial.assignment."direct"— trial was in the k-means fit (WT / PTZ / 4-AP); these animals define the prototypes."assigned"— trial projected onto the existing clustering by nearest centroid (the genetic edits); scored against prototypes, did not shape them.
- assignment
- clustering_run
- clustering_run_id
- trial
- trial_id
- class tadpose.database.ClusteringType(**kwargs)[source]
Bases:
BaseClustering configuration (e.g. ‘posture+velocity k=36’).
- clustering_type
- clustering_type_id
- clusterings
- class tadpose.database.DatabaseHandler(connection_string)[source]
Bases:
objectContext-managed SQLAlchemy session wrapper.
Creates tables and inserts static body-part rows when a new SQLite database is initialised.
Usage:
with DatabaseHandler("sqlite:///tadpoles.db") as db: db.add_record(Investigator(first_name="Alex", last_name="Matthews")) investigators = db.get_records(Investigator)
- Parameters:
connection_string (str)
- add_record(record)[source]
Insert a single record and commit.
- Parameters:
record (Base)
- Return type:
None
- find_series_by_attributes(attribute_ids, experiment_type_id, investigator_id, experiment_date)[source]
Find an ExperimentSeries matching all given attribute IDs.
Queries the many-to-many attributes relationship on the associated ExperimentType rather than hardcoded column slots.
- class tadpose.database.ExperimentSeries(**kwargs)[source]
Bases:
BaseA single session: one type, one investigator, one date.
- experiment_date
- experiment_type
- experiment_type_id
- investigator
- investigator_id
- series_id
- videos
- class tadpose.database.ExperimentType(**kwargs)[source]
Bases:
BaseProtocol definition (e.g. ‘4-AP dose-response’, ‘PTZ titration’).
- attributes
- experiment_attribute_1
- experiment_attribute_2
- experiment_attribute_3
- experiment_attribute_4
- experiment_attribute_5
- experiment_series
- experiment_type_id
- long_name
- protocol
- short_name
- class tadpose.database.ExperimentTypeAttribute(**kwargs)[source]
Bases:
BaseFree-form tag that can be attached to an ExperimentType.
- experiment_type_attribute_id
- experiment_types
- name
- class tadpose.database.FrameCluster(**kwargs)[source]
Bases:
BaseCluster label of one frame under one
ClusteringRun.Deliberately narrow — the big table.
trial_idandframe_numberare reached throughtime_series; the direct/assigned distinction throughClusteringRunTrial; so nothing is duplicated per frame. Alex’s legacy labels stay inclusteringfor the ARI cross-check.- clustering_run
- clustering_run_id
- frame_cluster_id
- label
- time_series
- time_series_id
- class tadpose.database.FrameClusterProportion(**kwargs)[source]
Bases:
BasePre-aggregated PM abundance per
(run, trial)— the fingerprint source.runs × trials × krows (thousands), not 64 M, so per-animal fingerprint queries never scanframe_cluster.n_framesis the raw count;proportion = n_frames / that trial's total labelled frames under the run(the value fingerprints use). A stored aggregate, so it is derived data: rebuilt fromframe_clusterafter each run, which stays the source of truth. Split by cohort viaclustering_run_trial.assignment.- clustering_run
- clustering_run_id
- frame_cluster_proportion_id
- label
- n_frames
- proportion
- trial
- trial_id
- class tadpose.database.Frog(**kwargs)[source]
Bases:
BaseParent female used for breeding.
- background_strain
- female_identifier
- female_tank
- frog_id
- tadpole_groups
- class tadpose.database.Investigator(**kwargs)[source]
Bases:
BaseResearcher who conducted the experiment.
- experiment_series
- first_name
- investigator_id
- last_name
- class tadpose.database.Posture(**kwargs)[source]
Bases:
BaseFrons-aligned body-part position at one frame.
- body_part
- body_part_id
- posture_id
- time_series
- time_series_id
- x_pos_mm
- y_pos_mm
- class tadpose.database.TadpoleGroup(**kwargs)[source]
Bases:
BaseClutch of tadpoles from a single fertilisation event.
- development_stage
- fertilisation_date
- mother
- mother_id
- seq_folder
- tadpole_group_id
- transgene
- trials
- class tadpose.database.TimeSeries(**kwargs)[source]
Bases:
BaseOne frame of one trial — the temporal backbone.
- clusterings
- frame_number
- postures
- time_series_id
- trajectories
- trial
- trial_id
- velocities
- class tadpose.database.Trajectory(**kwargs)[source]
Bases:
BaseRaw tracked position (pixels → mm) of one body part at one frame.
- body_part
- body_part_id
- time_series
- time_series_id
- trajectory_id
- x_pos_mm
- y_pos_mm
- class tadpose.database.Trial(**kwargs)[source]
Bases:
BaseOne tadpole in one well in one video.
- cluster_proportions
- run_trials
- tadpole_group
- tadpole_group_id
- time_series
- trial_id
- video
- video_id
- well_geometry
- well_number
- well_type
- well_type_id
- class tadpose.database.Velocity(**kwargs)[source]
Bases:
BaseBody-centric velocity at one frame.
- slip_mm_s
- thrust_mm_s
- time_series
- time_series_id
- velocity_id
- yaw_rad_s
- class tadpose.database.Video(**kwargs)[source]
Bases:
BaseOne video file from a Raspberry Pi camera session.
- camera
- date_time
- experiment_series
- filename
- fps
- pix2mm
- series_id
- trials
- video_id
- video_series_num
- video_series_size
- class tadpose.database.WellGeometry(**kwargs)[source]
Bases:
BasePer-well ring-CNN geometry: centre, radius, and pixel scale.
Keyed 1:1 on
trial_id— a trial already is one well of one video, sovideo_idandwell_numberare reached through the trial, not stored again here.pix2mm = 2 * r_px / 15.6(well diameter, caliper-confirmed). A position becomes well-centred mm via(x_px - cx_px) / pix2mm; thigmotaxis uses(cx_px, cy_px). Geometry source of truth, replacingvideo.pix2mm. (Empty wells carry no trial, hence no geometry row — they are never analysed.)- created
- cx_px
- cy_px
- detector
- pix2mm
- r_px
- trial
- trial_id
- class tadpose.database.WellType(**kwargs)[source]
Bases:
BaseExperimental condition applied to a well (drug, concentration).
- attributes
- description
- name
- trials
- well_attribute_1
- well_attribute_2
- well_attribute_3
- well_attribute_4
- well_attribute_5
- well_type_id
- class tadpose.database.WellTypeAttribute(**kwargs)[source]
Bases:
BaseFree-form tag for well conditions (e.g. ‘4-AP’, ‘10 mM’).
- name
- well_type_attribute_id
- well_types
Output paths and metadata file conventions.
Central owner of the per-run output directory layout and the metadata CSV/JSON file locations.
- class tadpose.file_manager.FileManager[source]
Bases:
object- create_subfolders()[source]
Creates predefined subdirectories within the base output folder. Does not raise an error if the directories already exist.
- setup_file_manager(base_output_path, db_file, video_folder, python_interpreter, dlc_config, script_base_path)[source]
Sets up and verifies all paths necessary for an experiment, including the database file, video file, and output directory. It initializes the experiment’s directory structure by creating subfolders for various data types.
- Parameters:
- Raises:
ValueError – If any required file or directory is not selected or does not exist.
The function updates internal dictionaries to manage paths efficiently: - file_dict to keep track of file locations like the database and video files. - path_dict to store paths to important directories and subdirectories, ensuring that all components of the experiment can reference these locations easily.
This method is typically called at the start of an experiment setup process to ensure all necessary files and folders are properly configured and exist.
Interactive 24-well plate assignment.
Console-driven assignment of well types and tadpole groups across the 24 wells of a plate.
- class tadpose.plate_manager.PlateManager(db_handler)[source]
Bases:
object- assign_tadpole_mother()[source]
Collects up to five mothers, allowing the user to choose from existing mothers, create new ones, or delete a selected mother. :returns: List of Experimentgroupmothers instances.
- collect_well_attributes()[source]
Collects up to five attributes, allowing the user to choose from existing attributes, create new ones, or delete a selected attribute. :returns: List of ExperimentTypeAttributes instances.
- confirm_action(prompt_message)[source]
Asks the user for confirmation before proceeding with an action.
- confirm_tadpole_group(mother, fertilisation_date, development_stage, seq_folder, transgene)[source]
Displays experiment details and asks the user for confirmation before proceeding.
- Returns:
True if the user confirms, False otherwise.
- Return type:
- confirm_well_type(name, attributes)[source]
Displays experiment details and asks the user for confirmation before proceeding.
- create_new_tadpole_mother()[source]
Prompts user to enter a name for a new mother and creates it. :returns: The new Experimentgroupmothers instance.
- create_new_well_attribute()[source]
Prompts user to enter a name for a new attribute and creates it. :returns: The new ExperimentTypeAttributes instance.
- delete_tadpole_group(tadpole_groups)[source]
Allows the user to delete a tadpole group from the selected list by specifying its index.
- delete_well_attribute(attributes)[source]
Allows the user to delete an attribute from the selected list by specifying its index.
- delete_well_type(well_types)[source]
Allows the user to delete a well type from the selected list by specifying its index.
- display_existing_tadpole_groups()[source]
Retrieves and formats a PrettyTable of all tadpole groups in the database with additional mother’s unique identifier, displaying until user decides to stop. :returns: Table containing the list of experiment groups. :rtype: PrettyTable
- display_existing_tadpole_mothers()[source]
Continuously retrieves and formats a PrettyTable of all mother types in the database until the user decides not to view any more protocols. :returns: Table containing the list of experiment types. :rtype: PrettyTable
- display_existing_well_type_attributes()[source]
Continuously retrieves and formats a PrettyTable of all well types in the database until the user decides not to view any more protocols. :returns: Table containing the list of experiment types. :rtype: PrettyTable
- display_existing_well_types()[source]
Continuously retrieves and formats a PrettyTable of all well types in the database until the user decides not to view any more protocols. :returns: Table containing the list of experiment types. :rtype: PrettyTable
- display_selected_tadpole_groups(tadpole_groups)[source]
Displays selected tadpole groups with their detailed information including mother’s unique ID.
- Parameters:
tadpole_groups (list) – A list of TadpoleGroup objects to display.
- Returns:
Outputs directly to console using PrettyTable.
- Return type:
None
- enter_new_well_type()[source]
Interactively creates a new experiment type and adds it to the database after user confirmation, including the possibility to add up to five attributes by choosing existing ones or creating new ones.
- enter_valid_date()[source]
Prompts the user for a date until a valid format (DD-MM-YYYY) is entered. :returns: The valid entered date. :rtype: datetime.date
- enter_valid_time()[source]
Prompts the user for a time and validates that it is in a correct format. :returns: The valid entered time. :rtype: datetime.time
- get_valid_development_stage()[source]
Prompts the user for a development stage and validates that it is an integer.
- initialise_plate_state()[source]
Initializes the plate state with lists for wells and tadpoles, each having 24 entries set to None.
- select_existing_tadpole_group()[source]
Allows the user to select an existing attribute by entering its ID. :returns: The selected ExperimentgroupAttributes instance or None if not found.
- select_existing_tadpole_mother()[source]
Allows the user to select an existing mother by entering its ID. :returns: The selected Experimentgroupmothers instance or None if not found.
- select_existing_well_attribute()[source]
Allows the user to select an existing attribute by entering its ID. :returns: The selected ExperimentTypeAttributes instance or None if not found.
- select_existing_well_type()[source]
Allows the user to select an existing attribute by entering its ID. :returns: The selected ExperimentTypeAttributes instance or None if not found.
- visualize_plate()[source]
Visualizes the current state of the plate using PrettyTable with cells labeled from 1 to 24, arranged in 6 columns and 4 rows, with lines between rows for clarity and no headers.
Interactive experiment and investigator setup.
Console-driven creation of experiment types, investigators and series records in the tadpole database.
- class tadpose.experiment_manager.ExperimentManager(db_handler)[source]
Bases:
objectManages the creation and management of experiments, including managing investigators and experiment types.
- db_handler
The database handler for accessing experiment-related data.
- Type:
- collect_attributes()[source]
Collects up to five attributes, allowing the user to choose from existing attributes, create new ones, or delete a selected attribute. :returns: List of ExperimentTypeAttributes instances.
- confirm_action(prompt_message)[source]
Asks the user for confirmation before proceeding with an action.
- confirm_experiment(short_name, attributes)[source]
Displays experiment details and asks the user for confirmation before proceeding.
- create_new_attribute()[source]
Prompts user to enter a name for a new attribute and creates it. :returns: The new ExperimentTypeAttributes instance.
- delete_attribute(attributes)[source]
Allows the user to delete an attribute from the selected list by specifying its index.
- display_selected_options(investigator_id, experiment_type_id)[source]
Displays the names of the selected investigator and experiment type. :param investigator_id: ID of the selected investigator. :type investigator_id: int :param experiment_type_id: ID of the selected experiment type. :type experiment_type_id: int
- display_tables_side_by_side()[source]
Displays investigators and experiment types side by side for easy comparison and selection. Uses padding to ensure both tables are aligned even if they have different numbers of rows.
- enter_new_experiment_type()[source]
Interactively creates a new experiment type and adds it to the database after user confirmation, including the possibility to add up to five attributes by choosing existing ones or creating new ones.
- enter_new_investigator()[source]
Interactively creates a new investigator and adds them to the database after user confirmation.
- manage_experiments()[source]
Main method to start the experiment management process. Allows the user to add, select investigators and experiment types, showing both tables side by side for easy reference. Returns a dictionary with tions(experimenthe selected investigator and experiment type IDs.
- select_existing_attribute()[source]
Allows the user to select an existing attribute by entering its ID. :returns: The selected ExperimentTypeAttributes instance or None if not found.
- select_experiment_type()[source]
Allows the user to select an experiment type by ID from a displayed list. :returns: The ID of the selected experiment type, or None if selection is invalid. :rtype: int
- select_investigator()[source]
Allows the user to select an investigator by ID from a displayed list. :returns: The ID of the selected investigator, or None if selection is invalid. :rtype: int
- show_experiment_types(do_print=True)[source]
Retrieves and formats a PrettyTable of all experiment types in the database.
- Parameters:
do_print (bool) – print the table here (the side-by-side view sets this False so the table is not also printed full-width on its own).
- Returns:
Table containing the list of experiment types.
- Return type:
PrettyTable
Save and reload experiment-setup presets.
Persists experiment / plate / camera setup choices so a session can be replayed without re-entering them.
Write pipeline results into the database.
Ingests per-trial trajectory, posture and velocity outputs and commits them to the tadpole database.
- class tadpose.result_manager.ResultManager(base_output_path, db_file, video_folder, python_interpreter, dlc_config, script_base_path, video_number)[source]
Bases:
object- check_files_loadable()[source]
Check if all files referenced in the metadata are loadable. If any files are not loadable, print the list of unloadable files and raise an error.
- enter_results()[source]
Process each row in the metadata DataFrame to create experiments, flies, and trials. Assumes that insert_experiment has already been called and set self.experiment_id.
- static parse_float_point_num(value)[source]
Parses the fly attribute value, converting NaN to None and ensuring the value is an integer.
- Parameters:
value – The value to parse.
- Returns:
The parsed value or None if the value is NaN.
- static parse_integer(value)[source]
Parses the fly attribute value, converting NaN to None and ensuring the value is an integer.
- Parameters:
value – The value to parse.
- Returns:
The parsed value or None if the value is NaN.
- remove_existing_video()[source]
Idempotent re-ingest: drop any prior ingest of THIS video.
Deletes the video’s trajectory / posture / velocity, its time_series, its trials and the video row (the shared ExperimentSeries is left for reuse). Without this a re-run would insert a duplicate video instead of replacing it – and a partial/aborted ingest would leave stale rows.
Build and submit the SLURM pipeline DAG.
Constructs the dependent SLURM job chain (split, track, extract, ingest) for the full per-plate workflow.
- class tadpose.slurm_jobs.SlurmJobManager(file_manager, meta_data_table, gpu_partition='aoraki_gpu_H100,aoraki_gpu_A100_80GB,aoraki_gpu_A100_40GB,aoraki_gpu_L40,aoraki_gpu_L4_24GB,aoraki_gpu_RTX3090')[source]
Bases:
object- create_slurm_script(script_parameters, mode='python')[source]
Generates a SLURM script file for a given analysis task using parameters from a dictionary.
- Parameters:
script_parameters (dict) – Parameters for the SLURM script including: - partition (str): The partition where the job should run. - filename (str): The filename for the SLURM script. - python_script (str): Path to the Python script to execute. - jobname (str): Name of the job. - memory (str): Memory allocation for the job. - script_variables (str): String with variables to pass to the Python script. - gpus_per_task (int): Number of GPUs per task. - nodes (int): Number of nodes to use. - ntasks_per_node (int): Number of tasks per node. - runtime_sec(num): How long the script can run at max in seconds
- create_sql_entry_slurm_script(individual_video_name, individual_video_duration, video_number, memory_GB_int=32, nodes=1, cpus_per_task=1, ntasks=1)[source]
- create_tracking_slurm_script(gpu_jobs, individual_video_name, video_duration, gpus_per_task=1, memory_GB_int=24, nodes=1, cpus_per_task=8, ntasks=1)[source]
- create_trajectory_extraction_slurm_script(individual_video_name, tracking_output_fileposition, well_num, video_duration, gpus_per_task=1, memory_GB_int=64, nodes=1, cpus_per_task=1, ntasks=1)[source]
- create_video_splitting_slurm_script(individual_raw_video_path, individual_video_name, individual_video_duration, memory_GB_int=32, nodes=1, cpus_per_task=16, ntasks=1)[source]
- format_duration_for_sbatch(duration_sec)[source]
Formats the duration in seconds to the SBATCH time format (D-HH:MM:SS).
- manage_workflow(num_wells=24, wait_on_before_sql_jobs=None, wait_on_job_before_start=None, gpu_chunk_size=1)[source]
Build and submit the split -> track -> extract -> ingest DAG.
Login-node pre-flight: the split and per-well track stages are the expensive ones (whole-plate CPU cut; one GPU allocation per well), and their completion artifacts are a cheap directory glob. So they are checked HERE, on the submit host, and finished units are never sent to SLURM – a resume of a mostly-done video costs zero split/track allocations instead of one no-op allocation per unit. The cheap CPU extract/ingest jobs still carry their own in-job resume gates (their completion lives inside a 78 MB fixed-format h5 / the DB, not cheaply interrogable from the login node), so they self-skip on the compute node.
- manage_workflow_without_splitting(num_wells=24, wait_on_job_before_start=None, gpu_chunk_size=1)[source]
Manages the full workflow of splitting, tracking, analyzing, and compiling results.
- manage_workflow_without_splitting_or_tracking(num_wells=24, wait_on_job_before_start=None, gpu_chunk_size=1)[source]
Manages the full workflow of splitting, tracking, analyzing, and compiling results.
- readd_rtx6000_to_pending()[source]
Re-add the gated RTX6000 partition to this user’s pending jobs.
A fresh submit silently strips RTX6000 (its AllowQos excludes our default normal,ondemand QOS). Queued GPU jobs are auto-elevated to gpu_unlimited, which IS allowed there, so
scontrol updateis now accepted – widening the node pool the 16-job cap can reach.- Return type:
None
- submit_job(script_path, dependency_id=None, dependency_type='afterok')[source]
Submit a job to SLURM with an optional dependency.
- dependency_type:
- ‘afterok’ – start only if the dependency SUCCEEDED (data deps:
extract needs the track’s h5, etc.).
- ‘afterany’ – start once the dependency FINISHES, success or fail
(serialisation only, e.g. the per-video ingest chain: one broken ingest must not block the rest).
Pull representative frames per tadpole.
Extracts and assembles example video frames for given tadpole and trial ids from the database.
- class tadpose.frame_getter.TadpoleFrameExtractor(data_csv_file, ids_from_np_data, db_path, base_video_dir)[source]
Bases:
object- overlay_body_parts(frame, body_parts)[source]
Overlays body parts on the given frame.
- Parameters:
frame (numpy.ndarray) – The video frame to plot on.
body_parts (list of tuples) – List of (x, y, body_part_id) representing body part coordinates and their IDs.