Detecting Attendance Outliers with a Z-Score in Python

This guide walks through one precise task: given a normalized attendance fact table, compute a per-student attendance-rate over a rolling window and flag the learners whose rate is a statistical outlier against their section cohort — first with a standard z-score, then with a robust MAD-based modified z-score that does not get blinded by the very outliers it is meant to find. It is the runnable companion to the attendance anomaly detection guide and assumes the canonical states produced by the attendance state normalization rules are already in place.

The reason this needs two statistics rather than one is that a classic z-score computes its own spread from the data it is scanning. A handful of chronically absent students inflate the standard deviation, widen the band, and hide the next struggling learner. The median and the median absolute deviation (MAD) do not move when a minority collapses, so the modified z-score keeps flagging the next student. The diagram below shows the same outlier under both tests.

Prerequisites

Confirm each of these before running the procedure — the script assumes all of them hold.

Python 3.10+ (the code uses str | None unions and modern type hints).
pandas==2.2.* and numpy==1.26.* installed in the virtualenv.
A normalized attendance export with the canonical states already applied — one row per (student_token, course_section_id, session_ts, canonical_state), as produced by the attendance state normalization rules. Raw vendor labels are not acceptable input here.
Student identifiers already tokenized to salted SHA-256 surrogates, or a salt available in ATTENDANCE_HASH_SALT if you must tokenize raw ids at read time. Never load raw student ids into the frame.
A defined rolling window length (this guide uses a trailing 28-day window) and a minimum session count below which a window is suppressed rather than scored.

Step-by-step implementation

1. Load the normalized frame and confirm the grain. Read the attendance events into a DataFrame and assert one row per (student_token, course_section_id, session_ts); duplicate session rows would double-count into the rate and manufacture false outliers.

2. Reduce canonical states to a present-fraction, excluding excused sessions. Map PRESENT and LATE to the numerator, ABSENT and UNEXCUSED to the denominator only, and drop EXCUSED and UNRECORDED from both — an excused or unmarked session must never count as an absence, the single rule that most affects fairness.

3. Cut the trailing rolling window per student. For each window_end, restrict to the prior 28 days and compute attendance_rate = present_or_late / recorded_sessions. Windows overlap intentionally so the score reflects a trend, not one noisy session.

4. Suppress sparse windows. Where recorded_sessions is below the minimum, mark the window low-confidence and exclude it from scoring; a two- or three-session window has no statistical power and one absence swings it wildly.

5. Compute the classic z-score within each section-window cohort. Standardize each learner’s rate against the mean and population standard deviation of their section for that window: $z = (x - \mu)/\sigma$ . This is the familiar baseline, kept for comparison.

6. Compute the robust modified z-score with median and MAD. Replace $\mu$ and $\sigma$ with the cohort median $\tilde{x}$ and the median absolute deviation, scaled by $0.6745$ :

$\text{MAD} = \operatorname{median}(\lvert x_i - \tilde{x}\rvert), \qquad M_i = \frac{0.6745\,(x_i - \tilde{x})}{\text{MAD}}$

Guard the degenerate case $\text{MAD}=0$ (more than half the cohort sharing one rate) so the test suppresses instead of dividing by zero.

7. Flag one-sided low outliers and record provenance. Attendance risk is one-sided — an unusually high rate is not a concern — so flag mod_z <= -3.5, falling back to z <= -3.0 only where MAD was degenerate, and record which test fired alongside the window_end so any flag is explainable to an advisor.

Complete runnable code block

python

import os
import hashlib
import numpy as np
import pandas as pd

WINDOW_DAYS = 28          # trailing rolling-window length
MIN_SESSIONS = 5          # windows thinner than this are suppressed
MOD_Z = 3.5               # |modified z| flag threshold
Z_HARD = 3.0              # plain z fallback threshold (used only if MAD == 0)
PRESENT_EQUIV = {"PRESENT", "LATE"}     # count toward the numerator
COUNTED = {"PRESENT", "LATE", "ABSENT", "UNEXCUSED"}  # in the denominator
SALT = os.environ.get("ATTENDANCE_HASH_SALT", "rotate-me-from-secrets").encode()


def tokenize(raw_id: str) -> str:
    """Salted SHA-256 — FERPA-safe; only used if ids arrive un-tokenized."""
    return "stu_" + hashlib.sha256(SALT + raw_id.encode()).hexdigest()[:24]


def window_rates(events: pd.DataFrame, window_end: pd.Timestamp) -> pd.DataFrame:
    """Attendance rate per (student, section) over the trailing window."""
    start = window_end - pd.Timedelta(days=WINDOW_DAYS)
    w = events[(events["session_ts"] > start) & (events["session_ts"] <= window_end)].copy()
    w = w[w["canonical_state"].isin(COUNTED)]  # excused/unrecorded excluded
    w["present"] = w["canonical_state"].isin(PRESENT_EQUIV).astype(int)
    agg = (
        w.groupby(["student_token", "course_section_id"])
         .agg(recorded_sessions=("present", "size"), present=("present", "sum"))
         .reset_index()
    )
    agg["attendance_rate"] = agg["present"] / agg["recorded_sessions"]
    agg["window_end"] = window_end
    return agg


def _modified_z(x: pd.Series) -> pd.Series:
    med = x.median()
    mad = (x - med).abs().median()
    if mad == 0:
        return pd.Series(np.nan, index=x.index)  # degenerate -> caller falls back
    return 0.6745 * (x - med) / mad


def flag_outliers(rates: pd.DataFrame) -> pd.DataFrame:
    df = rates.copy()
    trusted = df["recorded_sessions"] >= MIN_SESSIONS
    grp = df.groupby(["course_section_id", "window_end"])["attendance_rate"]
    df["z"] = grp.transform(lambda s: (s - s.mean()) / s.std(ddof=0))
    df["mod_z"] = grp.transform(_modified_z)

    def _test(row: pd.Series) -> str | None:
        if not row["_trusted"]:
            return None
        if pd.notna(row["mod_z"]) and row["mod_z"] <= -MOD_Z:
            return "mad_zscore"
        if pd.isna(row["mod_z"]) and pd.notna(row["z"]) and row["z"] <= -Z_HARD:
            return "zscore"  # MAD degenerate this window; fall back to plain z
        return None

    df["_trusted"] = trusted
    df["flag"] = df.apply(_test, axis=1)
    df["is_outlier"] = df["flag"].notna()
    return df.drop(columns="_trusted")


if __name__ == "__main__":
    events = pd.read_parquet(os.environ["ATTENDANCE_PARQUET"])
    events["session_ts"] = pd.to_datetime(events["session_ts"], utc=True)
    ends = pd.date_range(events["session_ts"].min().normalize(),
                         events["session_ts"].max().normalize(), freq="7D", tz="UTC")
    rates = pd.concat([window_rates(events, e) for e in ends], ignore_index=True)
    flagged = flag_outliers(rates)
    print(flagged[flagged["is_outlier"]]
          [["student_token", "course_section_id", "window_end",
            "attendance_rate", "mod_z", "flag"]].to_string(index=False))

Verification and output validation

Confirm the scanner behaves before you act on a single flag:

Grain is unique. assert not rates.duplicated(["student_token", "course_section_id", "window_end"]).any() — one score per student per section per window.
Excused never counts as absence. Build a fixture student with only EXCUSED rows in a window and assert they produce no recorded sessions and are absent from rates, not flagged.
Sparse windows are suppressed. Assert every is_outlier row has recorded_sessions >= MIN_SESSIONS; a flag on a thin window is a bug.
MAD guard holds. Feed a cohort where every rate is identical and assert mod_z is NaN and no divide-by-zero occurs.
One-sided. Assert no student with an above-cohort rate is ever flagged; a high attendance rate is not an anomaly.
Tokens only. assert flagged["student_token"].str.startswith("stu_").all() (or matches your surrogate format) so no raw identifier leaked into the output.

Troubleshooting

Every student in a small seminar gets flagged. The cohort is too small for a z-score to mean anything, so noise dominates. Raise MIN_SESSIONS, or pool the cohort to section-term instead of section-window; the suppression gate exists precisely for thin groups.
A punctual student flags as an outlier. LATE is being folded into the absence count upstream. Confirm the normalization layer keeps LATE distinct from ABSENT; the numerator here counts LATE as present-equivalent, and if the input already lost that distinction the fix belongs in attendance state normalization, not here.
RuntimeWarning: invalid value encountered in divide on the z column. A section-window has zero variance (std == 0), so every classic z is NaN. This is expected — the modified z with its MAD guard is the primary test; the plain z is a labeled fallback only.
The whole roster flags after a holiday week. Fixed 7-day window ends span a no-instruction week and depress rates uniformly. Cut window_end on the academic calendar and exclude holidays from the session set before aggregating.
A student on documented medical leave is flagged. Their absences arrived as ABSENT rather than EXCUSED, so the excused exclusion never applied. This is an upstream coding error; verify the excused-absence workflow wrote EXCUSED states before the export.
Flag counts differ between two runs on the same data. A non-deterministic window boundary (local time zone, or freq misaligned with the data) shifts sessions across window edges. Anchor all timestamps to UTC as shown and keep window_end generation deterministic.

Attendance Anomaly Detection — the parent layer that combines this z-score test with run and rate-of-change detectors.
Flagging Consecutive Absences in Attendance Data — the complementary run detector that catches a sudden stop a rolling rate still masks.
Attendance State Normalization Rules — the canonical states this scanner counts, and where tardy-versus-absent fidelity is enforced.
Early-Warning At-Risk Scoring — the downstream model that consumes these outlier flags as a behavioral feature.

Part of: Attendance Anomaly Detection