Behavioural capture stack

What is read from the video?

Inputs: Any modern video: venue AV, GoPro, iPhone, webcam. No wearables.
Computation: Frames are sampled, not read one by one, so compute stays bounded on a long session. A 2 minute clip runs at 12 frames a second, a 10 minute clip at 2, an hour at one frame per 4 seconds. What runs on each sampled frame depends on the read. A room read, which is what a single event clip gets by default, runs a tiled face detector over the frame and keeps a count and a coarse per-face orientation. It does not build a facial mesh, and it derives no action units: at the 25 to 35 pixel face sizes a room produces, a fine-grained facial read would be noise. A single-subject read is the path that runs MediaPipe FaceMesh at 468 landmarks per face and derives action units from it. Face embeddings are not stored on either path. Only the scoring numbers survive the pass.
Output: On a room read, one record per sampled frame holding the room count, how many of those faces were large enough to score, and the fraction oriented toward the stage. Oriented toward the stage is a geometric proxy off the detector keypoints, not a gaze measurement, and no row is ever tied to a person. On a single-subject read, one record per sampled frame carrying the action-unit strengths and head pose for the one tracked face.
References: MediaPipe FaceMesh, 468 landmarks (Google Research, 2019). Single-subject reads only; a room read does not use it.

Synchrony Score

Did the audience react together or one face at a time?

Inputs: Face arrays from each sampled frame. Faces are matched by slot index inside a frame; identity is not tracked across the room.
Computation: Rolling 30 second windows. In each window, correlate the smile-probability track of each pair of attending faces with at least 3 frames, using Pearson r. Report the aggregate mean pairwise r per window and for the session. An r of 1 means the pair moved together.
Output: One r per window, from -1 to 1. The UI shows (r + 1) / 2, on a 0 to 1 scale.
When it abstains: A window with fewer than two attending faces reports zero pairs, not an estimate.
References: Inter-subject facial response synchrony (PMC11109022, 2024)

What share of seconds did the room point at the stage?

Inputs: Face arrays from each sampled frame, with the attending flag set from head angle relative to a stage-facing camera. Head angle, not gaze.
Computation: Group faces by second. For each second, divide attending faces by faces detected. A second is held when that fraction reaches 0.6. Report the held rate, the mean fraction, and the longest unbroken run.
Output: An attention fraction per second. A held rate. The longest sustained run, in seconds. In code: attentionFraction, heldRate, longestSustainedSec.
References: Standard head-pose attention proxy (per-frame pitch/yaw thresholding).

Where did the audible reactions land and how long did they hold?

Inputs: The AudioPeak stream. Each peak carries a timestamp, an RMS loudness, how long it held above threshold, and a guessed type: laugh, applause, ovation, or reaction.
Computation: Count peaks by type. Rank them by duration and keep the top N. Bin into 5 second buckets, tracking weighted duration and peak amplitude. Report the 95th percentile amplitude, the level the loudest 5 percent of peaks pass.
Output: Counts by type, a sustained ranking, the 95th percentile amplitude, and a per-bin payload for the heat bar.
References: Affectiva audience response methodology (acoustic envelope segmentation).

Compared to the prior session, what moved?

Inputs: Two sessions for the same speaker, each with a SessionSummary and its engagement buckets.
Computation: Take the change in overall score, mean attention, and reaction totals. Compute Cohen's d across the bucket engagement scores on pooled standard deviation. That is the size of a change in standard deviations: meaningful at 0.5, a trend at 0.2, noise below.
Output: The change on each metric, Cohen's d, and the band it falls in.
When it abstains: A session needs at least two buckets. With one short session the panel shows no prior, not a number.
References: Cohen's d effect size convention (Cohen, 1988).