Attendance Anomaly Detection for LMS Data Pipelines

Once attendance has been collapsed into a single auditable vocabulary, the next analytical question is not what state is this record but which of these records is unusual. Attendance anomaly detection is the layer that turns a clean fact table into an early signal: a student whose participation has quietly collapsed, a section whose absence rate spiked the week after a holiday, a roster where tardies are being coded as absences by one instructor and not another. For EdTech engineers, institutional data analysts, and academic IT teams, this is where behavioral risk becomes measurable — and where a careless statistical choice can either flood advisors with false alarms or miss the withdrawing student entirely.

This page defines the detection layer that sits immediately downstream of the attendance state normalization rules. It assumes the canonical five-state model — PRESENT, LATE, ABSENT, EXCUSED, UNEXCUSED, with UNRECORDED as the pre-mark state — is already in place, because anomaly detection is only as trustworthy as the states it counts. Where normalization answers “what happened,” detection answers “is this out of pattern,” and its output flows onward into early-warning at-risk scoring as one behavioral feature among several. It is the analytical half of the gradebook and attendance normalization layer.

Entity Model: From Events to a Rolling Feature Grain

Anomaly detection does not operate on raw attendance events. It operates on a derived grain: one row per learner per section per rolling window, carrying the summary statistics that make an outlier test meaningful. The canonical attendance_event fact table records one status per session; the anomaly layer aggregates those events into a windowed feature row and then scores it. Keeping these two grains in separate tables is what prevents a subtle but common corruption — mixing per-session states with per-window rates in the same aggregation and double-counting the result.

The feature grain is keyed by (student_token, course_section_id, window_end), where window_end is the closing boundary of a fixed-length rolling window (commonly the trailing four academic weeks). Each row carries the metrics a detector consumes:

Two modeling decisions carry most of the weight. First, the rate numerator must be defined explicitly: EXCUSED sessions are typically removed from both numerator and denominator rather than counted as present, so an excused-heavy window does not masquerade as strong attendance. Second, windows overlap: a trailing four-week window recomputed weekly means each session contributes to several feature rows, which is intentional — it smooths the series and lets a detector see a trend rather than a single noisy session. The overlap also means the anomaly table is append-mostly and versioned by window_end, never updated in place.

Inputs and the Detection Contract

The detection layer has exactly one authorized input: the normalized attendance fact table, read after it has already crossed the tokenization boundary. There is no vendor API in this layer and there must not be one — reaching back to Canvas, Moodle, or Blackboard mid-scan would couple a statistical job to network latency and rate budgets it has no business owning. The contract is that a completed, validated normalization run publishes a fresh attendance_event partition, and the anomaly job treats that partition as immutable and self-sufficient.

Reading from the fact table rather than raw payloads has three consequences the detection code must respect. First, states are already canonical, so a detector counts ABSENT and UNEXCUSED toward the absence numerator without re-parsing vendor labels. Second, EXCUSED and UNRECORDED are not absences — an unmarked roster or a documented medical absence must not inflate an anomaly, which is the single most consequential rule in the whole layer. Third, timestamps are already UTC-anchored, so a rolling window can be cut on calendar boundaries without re-litigating the time-zone questions that attendance state normalization already resolved.

The job materializes a windowed feature frame from that input and scores each row. Because windows overlap, the same session influences several feature rows; the detection code must therefore be a pure function of the feature frame, with no hidden per-run state, so that re-materializing last month’s windows reproduces last month’s flags exactly.

Detection Methods

Three complementary detectors cover the failure shapes attendance actually exhibits. No single test is sufficient: a z-score catches a student who is a statistical outlier against peers, a run detector catches a sudden hard stop that a rate has not yet fully absorbed, and a rate-of-change detector catches a steep personal decline that is still above the cohort floor. Running them in parallel and recording which fired is what keeps the output interpretable for an advisor.

Z-score outlier detection

The classic test standardizes a learner’s window metric against the distribution of that metric across the comparison cohort — usually the section, or the section-term for larger populations. For a value $x$ drawn from a cohort with mean $\mu$ and standard deviation $\sigma$, the standard score is:

$z = \frac{x - \mu}{\sigma}$

A learner whose windowed attendance_rate yields $z \le -3$ sits three standard deviations below the cohort mean and is flagged. The z-score is cheap and familiar, but it has a well-known weakness for this domain: it is computed from the very outliers it is trying to find, so a handful of chronically absent students inflate $\sigma$ and mask a fourth. In a small section, one extreme value can shift $\mu$ enough to hide the next.

Modified z-score with MAD

The robust alternative replaces the mean and standard deviation with the median and the median absolute deviation (MAD), which are insensitive to a minority of extreme values. The MAD of a cohort is the median of the absolute deviations from the cohort median $\tilde{x}$:

$\text{MAD} = \operatorname{median}\bigl(\,\lvert x_i - \tilde{x}\rvert\,\bigr)$

The modified z-score scales each value against it, with the constant $0.6745$ making the statistic consistent with the standard normal for large samples:

$M_i = \frac{0.6745\,(x_i - \tilde{x})}{\text{MAD}}$

A common threshold flags $\lvert M_i \rvert > 3.5$. Because the median and MAD do not move when one or two students collapse, the modified z-score keeps flagging the next struggling learner instead of being blinded by the last one — which is why it is the default detector for cohort-relative attendance outliers. The one degenerate case to guard is $\text{MAD} = 0$, which happens when more than half the cohort shares an identical rate; the code must fall back to a mean-absolute-deviation scaling or suppress the test rather than divide by zero.

Consecutive-absence run detection

Cohort-relative tests miss a learner who was fine last month and then simply stopped attending, because a trailing rate averages the collapse against prior good weeks. A run detector reads the ordered session sequence directly and flags the longest unbroken stretch of absences. Encoding the sequence as runs and testing the maximum run length against a threshold $N$ (commonly three to five sessions) catches the hard stop days before the rate crosses a cohort floor. This is the detector that most directly serves an intervention workflow, and it is developed in full in flagging consecutive absences in attendance data.

Rate-of-change detection

The third detector is personal rather than cohort-relative: it compares a learner’s current-window attendance_rate to their own prior window. A drop larger than a threshold $\delta$ — say a fall of more than 0.30 in present-fraction between consecutive windows — flags a steep personal decline even when the absolute rate is still respectable. This catches the strong-start student whose engagement is deteriorating fast but who has not yet fallen below the cohort, and it is deliberately independent of the other two so that a decline and an outlier can be reported as distinct signals.

Compliance Constraints

An attendance anomaly flag is not a neutral analytical artifact. It is a derived education record under FERPA, and it is a more sensitive one than the raw states it summarizes, because a flag is designed to trigger action: an advisor outreach, a financial-aid satisfactory-academic-progress review, a retention intervention. A false flag can therefore cause concrete harm, and a leaked flag exposes a behavioral inference about a named student. The compliance posture must reflect that elevated sensitivity.

Three rules are non-negotiable for this layer. First, every detector operates on student_token only — the salted SHA-256 surrogate produced upstream by cross-LMS student ID mapping — so no raw identifier ever enters a feature frame, a log line, or a flag row. Second, flags carry their provenance: each anomaly row records which detector fired, the threshold in force, and the window_end, so a student or advisor challenging a flag can see exactly why it was raised without the pipeline re-deriving it from scratch. Third, data minimization extends to the reason codes — the flag table stores that a run detector fired, not the free-text absence justifications behind those sessions, which may reference protected medical or disciplinary information and belong only in the source system. Re-identification of a token to contact a student happens through a separately access-logged service, never inside the scoring job.

Reference Python Implementation

The scanner below reads a normalized attendance feature frame and applies all three detectors, returning a flag frame keyed by the same grain. It is framework-light, uses Python 3.10+ syntax, and operates only on tokens. Cohort statistics are computed per section, the modified z-score guards the zero-MAD degenerate case, and windows too sparse to trust are suppressed rather than scored.

import numpy as np
import pandas as pd

MIN_SESSIONS = 5          # windows thinner than this are not trustworthy
Z_HARD = -3.0             # standard z threshold (present-rate is one-sided low)
MOD_Z = 3.5               # |modified z| flag threshold
RUN_N = 3                 # consecutive-absence run length that flags
RATE_DROP = 0.30          # window-over-window present-fraction fall that flags


def _modified_z(x: pd.Series) -> pd.Series:
    """Robust z using median and MAD; NaN where MAD is degenerate (0)."""
    med = x.median()
    mad = (x - med).abs().median()
    if mad == 0:
        return pd.Series(np.nan, index=x.index)  # caller suppresses, never /0
    return 0.6745 * (x - med) / mad


def scan_attendance(feat: pd.DataFrame) -> pd.DataFrame:
    """Flag anomalies on a per-(student_token, section, window_end) frame.

    Expected columns: student_token, course_section_id, window_end,
    sessions_in_window, attendance_rate, max_consecutive_absences, rate_delta.
    """
    df = feat.copy()
    trusted = df["sessions_in_window"] >= MIN_SESSIONS

    # Cohort-relative tests: computed within each section+window group.
    grp = df.groupby(["course_section_id", "window_end"])["attendance_rate"]
    df["z"] = grp.transform(lambda s: (s - s.mean()) / s.std(ddof=0))
    df["mod_z"] = grp.transform(_modified_z)

    def _flags(row: pd.Series) -> list[str]:
        if not row["_trusted"]:
            return []  # sparse window -> suppressed, not scored
        hits: list[str] = []
        if pd.notna(row["mod_z"]) and row["mod_z"] <= -MOD_Z:
            hits.append("mad_zscore")
        elif pd.notna(row["z"]) and row["z"] <= Z_HARD:
            hits.append("zscore")  # fall back to plain z only if MAD degenerate
        if row["max_consecutive_absences"] >= RUN_N:
            hits.append("run")
        if row["rate_delta"] <= -RATE_DROP:
            hits.append("rate_drop")
        return hits

    df["_trusted"] = trusted
    df["anomaly_flags"] = df.apply(_flags, axis=1)
    df["is_anomaly"] = df["anomaly_flags"].str.len() > 0
    out_cols = [
        "student_token", "course_section_id", "window_end",
        "attendance_rate", "z", "mod_z", "max_consecutive_absences",
        "rate_delta", "anomaly_flags", "is_anomaly",
    ]
    return df.loc[:, out_cols]

The scanner is a pure function of the feature frame, so replaying an earlier window reproduces its flags exactly, and the anomaly_flags list makes the output self-explaining: a row tagged ["mad_zscore", "run"] is both a cohort outlier and mid-collapse, a stronger signal than either alone. That list is precisely the shape the early-warning at-risk scoring layer consumes as a behavioral feature.

Failure Modes and Edge Cases

Attendance anomaly detection fails in ways that generic outlier tooling does not anticipate, because the academic calendar and human coding habits inject structure that violates the i.i.d. assumptions behind a z-score.

• Seasonality and holidays. A campus-wide holiday week depresses attendance for everyone, and a naive rate-of-change detector flags the entire roster. Cut windows on the academic calendar, exclude no-instruction days from the denominator, and compare a window against the same relative point in the term rather than raw wall-clock weeks. A cohort-relative modified z-score is naturally resistant here because the whole cohort moves together, which is one more reason to prefer it over an absolute-threshold rule.
• Sparse sections. A seminar of six students or a window with three recorded sessions produces cohort statistics with almost no power; one absence swings the mean wildly. The MIN_SESSIONS gate and the suppressed low-confidence path exist for exactly this, and the modified z-score’s zero-MAD guard covers the case where a tiny cohort shares one rate. Never emit an anomaly from a window you would not defend to the flagged student.
• Tardy-versus-absent conflation. If normalization or an instructor codes LATE as ABSENT, the absence numerator inflates and a punctual-but-tardy student is flagged as disappearing. This is not a detection bug but an upstream one; the detector inherits whatever the attendance state normalization rules produced, which is why the states must be trustworthy before any statistic is computed. Keep LATE distinct from ABSENT in the numerator and let the rate-of-change detector, not a run detector, carry chronic tardiness.
• Alert fatigue. A layer that flags too readily trains advisors to ignore it, which is worse than not flagging at all. Requiring agreement between detectors before escalation (an outlier and a run), tuning thresholds against a labeled term of known outcomes, and reporting a flag’s provenance so a human can dismiss it in seconds are the defenses. The goal is a small number of high-precision flags an advisor trusts, not a firehose.
• Excused absences read as risk. A documented medical leave is a run of EXCUSED sessions; counting it as an absence run manufactures a false crisis for a student already navigating a hardship. Excused states must be removed from every numerator and from run detection, and this is worth an explicit assertion in the scanner’s tests.

Coupling three complementary detectors with a suppression gate, calendar-aware windows, and provenance-rich flags gives an institution an attendance signal that is sensitive to real disengagement, resistant to the structural quirks of academic data, and defensible under the FERPA scrutiny that any intervention-triggering record must withstand.

Related

• Detecting Attendance Outliers with a Z-Score in Python — a runnable rolling-window z-score and MAD-based modified z-score scanner with tokenized ids.
• Flagging Consecutive Absences in Attendance Data — run-length encoding over sorted session records to flag runs of N consecutive absences.
• Attendance State Normalization Rules — the canonical five-state fact table this detection layer reads and depends on.
• Weighted Grade Calculation Engines — the sibling normalization layer that turns participation into a deterministic grade coefficient.
• Early-Warning At-Risk Scoring — the downstream consumer that folds these anomaly flags into a composite student-risk signal.

Part of: Gradebook & Attendance Normalization